Systems and methods for operating an input device in an augmented/virtual reality environment

ABSTRACT

In some embodiments, a system comprising one or more processors configured to track a location of an input device within a physical environment via a three-dimensional (3D) tracking system, and modify a tracking parameter of the 3D tracking system while tracking the location of the input device based on the determined location of the input device within the physical environment. The input device may be coupled to a virtual reality display system and tracking the location of the location of the input device can be used for interacting with the virtual reality display system.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a bypass continuation of PCT Application No.PCT/US2017/064080 entitled “A SYSTEM FOR IMPORTING USER INTERFACEDEVICES INTO VIRTUAL/AUGMENTED REALITY” filed Nov. 30, 2017, which is anon-provisional of and claims the benefit of priority of U.S.Provisional Application No. 62/428,276, entitled, “A SYSTEM FORIMPORTING USER INTERFACE DEVICES INTO VIRTUAL/AUGMENTED REALITY,” filedon Nov. 30, 2016, U.S. Provisional Application No. 62/428,241, entitled,“AUGMENTED/MIXED REALITY PERIPHERAL DISCOVERY AND USAGE SYSTEM,” filedon Nov. 30, 2016, U.S. Provisional Application No. 62/428,248, entitled,“SYSTEM FOR OBJECT CONTROL AND MANIPULATION IN VIRTUAL REALITY,” filedon Nov. 30, 2016, U.S. Provisional Application No. 62/428,253, entitled,“SYSTEM FOR NATURAL HAND INTERACTION WITH VIRTUAL OBJECTS,” filed onNov. 30, 2016, U.S. Provisional Application No. 62/428,488, entitled,“TRACKING SYSTEM FOR REAL OBJECTS IN VIRTUAL REALITY,” filed on Nov. 30,2016, and U.S. Provisional Application No. 62/428,339, entitled, “BLINDTEXT AND CONTROL INPUT SYSTEM FOR VIRTUAL REALITY,” filed on Nov. 30,2016 and a bypass continuation of PCT Application No. PCT/US2017/061649entitled “A SYSTEM FOR IMPORTING OBJECTS AS INPUT DEVICES INTOVIRTUAL/AUGMENTED REALITY” filed Nov. 14, 2017, which is anon-provisional and claims the benefit of priority of U.S. ProvisionalApplication No. 62/421,910, entitled, “A SYSTEM FOR IMPORTING OBJECTS ASINPUT DEVICES INTO VIRTUAL/AUGMENTED REALITY,” filed on Nov. 14, 2016.The disclosures of these applications are hereby incorporated byreference in their entirety for all purposes.

TECHNICAL FIELD

The present disclosure relates generally to virtual reality systems andin particular to providing user input devices and other peripherals in avirtual reality environment.

BACKGROUND

Virtual, mixed, or augmented reality can be associated with a variety ofapplications that comprise immersive, highly visual, computer-simulatedenvironments. These environments can simulate a physical presence of auser in a real or imagined world. The computer simulation of theseenvironments can include computer rendered images, which can bepresented by means of a graphical display. The display can be arrangedas a head mounted display (HMD) that may encompass all or part of auser's field of view.

A user can interface with the computer-simulated environment by means ofa user interface device or peripheral device, examples of which includea keyboard, game controller, joystick, mouse, stylus, speaker, andsteering wheel. When immersed in the computer-simulated environment, theuser may perform complex operations associated with the interfacedevice, including actuation of one or more keys or other input elementsof the interface device. However, there is need for improvement whenoperating within virtualized environments using physical interfacedevices, especially when performing productivity-based tasks.

SUMMARY

In certain embodiments, a system comprising one or more processors canbe configured to: track a location of an input device within a physicalenvironment via a three-dimensional (3D) tracking system, wherein theinput device is coupled to a virtual reality display system and whereinthe tracking the location of the location of the input device is usedfor interacting with the virtual reality display system; and modify atracking parameter of the 3D tracking system while tracking the locationof the input device based on the tracked location of the input devicewithin the physical environment. The tracking parameter can be furthermodified based on a biometric model of a user of the input device. Thebiometric model can be used by the one or more processors to determine alocation of the input device corresponding to the user holding the inputdevice with an appendage. Modifying the tracking parameter of the inputdevice can further include: tracking the location of the input deviceaccording to a first tracking accuracy profile in response todetermining that the input device is operating “in-air” in 3D space; andtracking the location of the input device according to a second trackingaccuracy profile in response to determining that the input device isoperating along a two-dimensional (2D) surface. The two-dimensionalsurface can be a physical surface. In some aspects, the first trackingaccuracy profile has a lower tracking accuracy than a tracking accuracyof the second tracking accuracy profile.

In certain embodiments, modifying the tracking parameter of the inputdevice can further include: determining that the input device is at alocation indicative of the input device being “in-air” in 3D space;determining that the input device is at a location indicative of a useroperating the input device “in-air” in 3D space and bracing their armagainst a surface; and tracking the location of the input deviceaccording to a third accuracy tracking profile in response todetermining that the input device is at a location indicative of theinput device being operating against a physical surface. The firsttracking accuracy profile can have a lower tracking accuracy than atracking accuracy of the second tracking accuracy profile, wherein thethird tracking accuracy profile has a higher tracking accuracy than thetracking accuracy of the second tracking accuracy profile, and whereinthe third tracking accuracy profile has a lower tracking accuracy thanthe tracking accuracy of the second tracking accuracy profile.Determining that the input device is at a location indicative of a useroperating the input device “in-air” in 3D space and bracing their armagainst a surface can include at least one of: using an imaging sensorto detect an orientation of the input device or the user; using abiomechanical model of the user to determine a location indicative of aportion of user being in a determined position while operating the inputdevice; or detecting micro-tremors of the input device induced by theuser while the input device is operated, where micro-tremors at orgreater than a threshold amplitude are indicative of the input deviceoperating “in-air” and the user operating the input device withoutbracing their arm against the surface, and where micro-tremors below thethreshold amplitude are indicative of the input device operating“in-air” and the user operating the input device bracing their armagainst the surface.

In some aspects, the input device can be of a list including: a stylus;a mobile device; a computer mouse; a presenter device; and a wearabledevice. The 3D tracking system may track in three axes in a Cartesiancoordinate system including a first, second, and third axis, and inresponse to determining that the input device is operating along the 2Dsurface, the one or more processors may be further configured to:determine which of the first, second, and third axes substantiallydefine a contour of the 2D surface; and suspend tracking along any ofthe first, second, and third axes that does not substantially define thecontour of the 2D surface while the input device is determined to beoperating along the 2D surface. The one or more processors can furtherbe configured to: determine a distance from the input device to aperipheral device, the input device and the peripheral device beingseparate and communicatively coupled to the same computer system; andmodify a tracking precision of the input device while tracking thelocation of the input device based on the determined distance from theinput device to the peripheral device. In some cases, the one or moreprocessors are configured to: modify the tracking precision of the inputdevice according to a first tracking sensitivity profile in response todetermining that the input device is within a threshold distance fromthe peripheral device; and modify the tracking precision of the inputdevice according to a second tracking sensitivity profile in response todetermining that the input device is not within a threshold distancefrom the peripheral device. The first tracking sensitivity profile canhave as a higher precision measurement than a precision measurement ofthe second tracking sensitivity profile. The one or more processors maybe configured to: change an operation of the input device based on thetracked location of the input device within the physical environment. Insome implementations, modifying the tracking parameter of the inputdevice can be based on the tracked location of the input device withinthe physical environment causes an operation including one of a list ofoperations including: changing a function of one or more buttons on theinput device based on a contextual usage of the input device; changingan operation of the input device in response to the input device beingmoved to a same location as a virtual object or physical object;changing a visual presentation of a virtual feature of the input devicein response to the input device being moved to a same location as thevirtual object the physical object; and initiating a haptic stimulus bya haptic device coupled to the input device in response to the inputdevice being moved to a same location as the virtual the physicalobject.

In some embodiments, a computer implemented method includes: tracking alocation of an input device within a physical environment via athree-dimensional (3D) tracking system, wherein the input device iscoupled to a virtual reality display system and wherein the tracking thelocation of the location of the input device is used for interactingwith the virtual reality display system; and modifying a trackingparameter of the 3D tracking system while tracking the location of theinput device based on the tracked location of the input device withinthe physical environment. The tracking parameter can be further modifiedbased on a biometric model of a user of the input device. The biometricmodel can be used by the one or more processors to determine a locationof the input device corresponding to the user holding the input devicewith an appendage.

In certain embodiments, modifying the tracking parameter of the inputdevice can further include: tracking the location of the input deviceaccording to a first tracking accuracy profile in response todetermining that the input device is operating “in-air” in 3D space; andtracking the location of the input device according to a second trackingaccuracy profile in response to determining that the input device isoperating along a two-dimensional (2D) surface. In some cases, thetwo-dimensional surface can be a physical surface. The first trackingaccuracy profile can have a lower tracking accuracy than a trackingaccuracy of the second tracking accuracy profile. Modifying the trackingparameter of the input device can further include: determining that theinput device is at a location indicative of the input device being“in-air” in 3D space; determining that the input device is at a locationindicative of a user operating the input device “in-air” in 3D space andbracing their arm against a surface; and tracking the location of theinput device according to a third accuracy tracking profile in responseto determining that the input device is at a location indicative of theinput device being operating against a physical surface.

In some embodiments, the first tracking accuracy profile can have alower tracking accuracy than a tracking accuracy of the second trackingaccuracy profile, wherein the third tracking accuracy profile has ahigher tracking accuracy than the tracking accuracy of the secondtracking accuracy profile, and wherein the third tracking accuracyprofile has a lower tracking accuracy than the tracking accuracy of thesecond tracking accuracy profile. In some cases, determining that theinput device is at a location indicative of a user operating the inputdevice “in-air” in 3D space and bracing their arm against a surface caninclude at least one of: using an imaging sensor to detect anorientation of the input device or the user; using a biomechanical modelof the user to determine a location indicative of a portion of userbeing in a determined position while operating the input device; ordetecting micro-tremors of the input device induced by the user whilethe input device is operated, where micro-tremors at or greater than athreshold amplitude are indicative of the input device operating“in-air” and the user operating the input device without bracing theirarm against the surface, and where micro-tremors below the thresholdamplitude are indicative of the input device operating “in-air” and theuser operating the input device bracing their arm against the surface.

In certain embodiments, a method of operating a virtual workstation caninclude determining a location of a physical peripheral input devicewithin a physical environment and determining, based on the location ofthe physical peripheral input device within the physical environment, anorientation of a first virtual display to render to a user of thephysical peripheral device, where content presented to the user via therendered first virtual display can be modifiable in response to the useroperating the physical peripheral input device. In some cases, thedetermined orientation of the first virtual display to render can beconfigured to remain at a fixed, perceived spatial relationship withrespect to the physical peripheral input device as the physicalperipheral input device moves within the physical environment. Incertain embodiments, the determined orientation of the first virtualdisplay to render can be configured to remain at a fixed, perceivedspatial relationship with respect to the physical peripheral inputdevice as the physical peripheral input device moves within the physicalenvironment in response to a movement of the physical input device froman initial location that is greater than a threshold distance, and thedetermined orientation of the first virtual display to render can beconfigured to remain in place in response to a movement of the physicalinput device from the initial location that is less than the thresholddistance.

In some embodiments, the method may further include determining, basedon the location of the physical peripheral input device within thephysical environment, an orientation of an interactive virtual displayto render to the user of the physical peripheral device, where contentpresented to the user via the rendered interactive virtual display canbe selectable by the user to modify an operation corresponding to use ofthe physical peripheral input device by the user, and where theinteractive virtual display can be rendered to appear to the user to bein proximity to the physical peripheral device. The operationcorresponding to use of the physical peripheral input device by the usermay include changing a command corresponding to actuation of an inputelement of the physical peripheral input device, changing an appearanceof the physical peripheral input device, an appearance of an inputelement of the physical peripheral input device, and/or changing arelationship between one or more rendered virtual displays and thephysical peripheral input device.

In further embodiments, the relationship between the one or morerendered virtual displays and the physical peripheral input device mayinclude selecting which of the one or more rendered virtual displaysinclude content that is modifiable in response to the user operating thephysical peripheral input device, selecting which of the one or morerendered virtual displays are anchored in position with respect to thephysical peripheral input device, and/or selecting how one of the one ormore rendered virtual displays are anchored in position with respect tothe physical peripheral input device including: a threshold distancethat the physical peripheral input device can be moved before the one ofthe one or more rendered virtual displays moves in response to thephysical peripheral input device being moved; or whether the one or morerendered virtual displays move in response to the physical peripheralinput device being moved. In some cases, the orientation of theinteractive virtual display can be rendered to appear to the user to beat a location that is integrated on or within a periphery of thephysical peripheral input device, or the interactive virtual display canbe rendered to appear to the user to be located on an area of thephysical peripheral input device including features that supplementfunctionality provided by the interactive virtual display.

In some embodiments, the features may include haptic feedback generatedby a haptic feedback generator integrated within the physical peripheralinput device to provide haptic feedback in response to user interactionwith the interactive virtual display. The features can include aphysical input element and where the content of the interactive virtualdisplay is rendered to appear to the user to be selectable by the userby actuation of the physical input element. The interactive virtualdisplay can be rendered to appear to the user to be located in an areain proximity to the physical peripheral input device. The content can bedetermined to be selected by the user, without physically contacting thephysical peripheral input device, by a sensor of the physical peripheralinput device. The method may further include determining a location of asecond physical peripheral input device within the physical environmentand determining, based on the location of the second physical peripheralinput device within the physical environment, an orientation of a secondinteractive virtual display to render to a user of the second physicalperipheral device. In such cases, the content presented to the user viathe rendered display can be selectable by the user of the secondphysical peripheral input device to modify an operation to use of thesecond physical peripheral input device by the user of the secondphysical peripheral input device, and the second interactive virtualdisplay can be rendered to appear to the user of the second physicalperipheral input device to be in proximity to the second physicalperipheral input device. In certain embodiments, the interactive virtualdisplay can be rendered to appear to the user to be at a spatiallocation determined, at least in part, based on a biomechanical model ofthe user to enable the user to reach the interactive display with anappendage.

In further embodiments, the method may include generating control datato cause a head-mounted display device to render the first virtualdisplay. The physical peripheral input device can be a computerkeyboard, mini-tower, or other peripheral that is preferably stationaryduring use. The orientation of the first virtual display may bedetermined further based on physical characteristics of an environmentthat the physical peripheral input device is located within. Thephysical characteristics may include barriers that would block a line ofsight of the user, and may be determined, in part, by a sensor of thephysical peripheral input device. In some cases, the first virtualdisplay can be rendered in response to actuation of a physical inputelement of the physical peripheral input device, in response to adetermined proximity of the user to the physical peripheral inputdevice, or in response to the physical peripheral input device beingplaced upon a working surface. Determination of the physical peripheralinput device being placed upon the working surface may be based on areading of a sensor attached to the physical peripheral input device.

In certain embodiments, the first virtual display can be rendered at afirst orientation such that the first virtual display appears to theuser to be at a first spatial orientation with respect to the physicalperipheral input device in response to a determination that the physicalperipheral input device is in a first state indicative of the useroperating the physical peripheral device and the first virtual displaymay not be rendered at the first orientation in response to adetermination that the physical peripheral input device is not in thefirst state. In some cases, the first virtual display can be rendered ata second orientation such that the first virtual display appears to theuser to be at a second spatial orientation with respect to the physicalperipheral input device in response to a determination that the physicalperipheral input device is not in the first state. The first virtualdisplay may be automatically rendered at the first orientation inresponse to a determination that the physical peripheral input device isin the first state. The physical peripheral input device may includefeatures for determining the location of the physical peripheral inputdevice within the physical environment by a tracking system and thetracking system can be used to determine an orientation of a physicaldisplay used to render the first virtual display. In some embodiments,the features can be selected from a list including a sensor, an emitter,and a marking. The sensor can be configured to detect or the emitter canbe configured to emit: visible light, infrared light, ultrasound,magnetic fields, or radio waves. In some cases, the physical peripheralinput device can include a plurality of the features to enable thephysical peripheral input device to be tracked within the physicalenvironment by any one of a plurality of tracking techniques. Thephysical peripheral input device may include an inertial measurementunit (IMU) and the location of the physical peripheral input devicewithin the physical environment can be determined using the IMU. Incertain embodiments, the orientation of the first virtual display can bedetermined based on a determined identity of the user of the physicalperipheral input device and wherein the orientation of the first virtualdisplay would be rendered differently for a differently identified user.

In some embodiments, a method of operating a peripheral device includesreceiving polling data from one or more sensors, the polling datacorresponding to physical characteristics of a physical environmentaround the peripheral device, determining an area to orient a virtualdisplay relative to the peripheral device within the physicalenvironment based on the physical characteristics, determining a spatialrelationship between the peripheral device and the projected virtualdisplay, and generating control data configured to cause an AR/VR-basedhead-mounted display (HMD) to project the virtual display in thedetermined area at a maintained spatial relationship between theperipheral device and the projected virtual display as the peripheraldevice is moved within the physical environment. The method can furtherinclude detecting that the peripheral device is placed on a surface orinterfaced by a user, where receiving the polling data from the one ormore sensors may occur in response to detecting that the peripheraldevice is placed on the surface or interfaced by the user. The methodcan include determining that the peripheral device is lifted off of thesurface, and generating second control data to cause the HMD to changethe spatial relationship between the peripheral device and the projectedvirtual display such that a volumetric area occupied by the peripheraldevice and the projected virtual display is reduced.

In some implementations, the method may further include determining asecond area to orient a virtual interactive display relative to theperipheral device, where the interactive display can be configured tofacilitate an augmentation of functional capabilities of the peripheraldevice, and determining a spatial relationship between the peripheraldevice and the projected interactive display, where the control data canbe further configured to cause the HMD to project the interactivedisplay in the determined second area and at a maintained spatialrelationship between the peripheral device and the projected interactivedisplay as the peripheral device is moved in the physical environment.The control data may cause the spatial relationship between theperipheral device and the virtual display to be maintained such that amovement of the peripheral device that is within a threshold distancefrom an initial location of the peripheral device does not cause thevirtual display to move and a movement of the peripheral device that isgreater than the threshold distance from the initial location of theperipheral device can cause the spatial relationship between theperipheral device and the projected interactive display to be fixed,where the spatial relationship between the peripheral device and theprojected interactive display is fixed as the peripheral device is movedin the physical environment.

According to some embodiments, the method can include determining athird area on the peripheral device to orient a virtual overlay, wherethe virtual overlay can be configured to further facilitate theaugmentation of the functional capabilities of the peripheral device,and determining a spatial relationship between the peripheral device andthe projected virtual overlay, where the control data is furtherconfigured to cause the HMD to project the virtual overlay in thedetermined third area and at a maintained spatial relationship betweenthe peripheral device and the projected interactive display as theperipheral device is moved in the physical environment.

In further embodiments, a hub device for interfacing physical peripheraldevices for a virtual environment includes a transceiver configured tocommunicate with a display system for presenting a virtual reality oraugmented reality virtual reality environment to a user, a trackingsubsystem configured to sense a location of a physical peripheral inputdevice within a physical environment, and one or more processors coupledto the transceiver and the tracking subsystem, the one or moreprocessors configured to determine, via the tracking subsystem, thelocation of the physical peripheral input device within the physicalenvironment, and transmit, via the transceiver, to the display system,the location of the physical peripheral input device within the physicalenvironment. In some cases, the tracking subsystem can sense thelocation of the physical peripheral input device using a techniquedifferent from a technique used by the display system to track alocation of the user or a head mounted display (HMD) worn by the user.The technique used by the tracking subsystem to track the physicalperipheral input device and the technique used by the display system totrack the location of the user or the HMD may be each selected from alist comprising: an ultrasonic emitter; an ultrasonic receiver; avisible light optical sensor; a visible light optical emitter; anon-visible light optical sensor; a non-visible light optical emitter; amagnetic field generator; a magnetic field sensor; a radio wave emitter;and a radio wave receiver. In some cases, the hub device can furthercomprising features for determining the location of the device withinthe physical environment by a system used to track a location of theuser or a head mounted display (MHD) worn by the user. The features maybe selected from a list including: a sensor; an emitter; and a marking,for example. The sensor can be configured to detect or the emitter isconfigured to emit: visible light, infrared light, ultrasound, magneticfields, or radio waves.

In some embodiments, the one or more processors of the hub device can beconfigured to: determine, via the tracking subsystem, the location ofthe physical peripheral input device in relation to the device;transform the location of the physical peripheral input device from acoordinate system used by the device to a coordinate system used by thedisplay system; and transmit, via the transceiver, to the displaysystem, the location of the physical peripheral input device using thecoordinate system of the display system. The coordinate system used bythe display system may be used to track a location of the user or anHMD, or track a location of the user or an HMD. In some cases, the oneor more processors of the hub device can be configured to receive, fromthe physical peripheral input device, a signal corresponding toactuation of an input element of the peripheral input device andtransmit, via the transceiver, to the display system, a signalcorresponding to the actuation of the input element.

In certain embodiments, the tracking subsystem of the hub device may beconfigured to sense locations each of a plurality of physical peripheralinput devices within the physical environment; and the one or moreprocessors can be configured to determine, via the tracking subsystem,the locations each of the plurality of physical peripheral input deviceswithin the physical environment and transmit, via the transceiver, tothe display system, the locations of each of the plurality of physicalperipheral input devices within the physical environment. The hub devicecan further comprising a memory storing a plurality of profiles, each ofthe plurality of profiles corresponding to a different physicalperipheral input device.

One of the plurality of profiles may include information for translatinga signal received from a physical peripheral input device correspondingto the one of the plurality of profiles to a signal interpretable by thedisplay system. The signal received from the physical peripheral inputdevice corresponding to the one of the plurality of profiles maycorrespond to actuation of an input element of the physical peripheralinput device corresponding to the one of the plurality of profiles. Insome cases, one of the plurality of profiles can include information forrendering a visual representation of a physical peripheral input devicecorresponding to the one of the plurality of profiles, or informationindicating features of a physical peripheral input device correspondingto the one of the plurality of profiles, where the features indicatecapabilities of the physical peripheral input device to supportaugmented virtual content.

In some embodiments, the features indicate an area of the physicalperipheral input device can include physical elements where augmentedvirtual content can be displayed to the user and wherein the elementssupport the augmented virtual content by providing a physical stimulusto a user interacting with the augmented virtual content or a tactileinput element for interacting with the augmented virtual content. Thelocation of the physical peripheral input device can be determinedwithin a three-dimensional space. In some cases, the tracking subsystemmay include, for determining a location of the physical peripheral inputdevice, at least two of: an ultrasonic emitter or receiver; an opticalsensor or receiver; a magnetic field generator or sensor; and a radiowave emitter or receiver.

According to certain embodiments, a hub device for determining a virtualworkstation for use with a virtual environment can include a transceiverconfigured to communicate with a display system for presenting a virtualreality or augmented reality virtual reality environment to a user; oneor more sensors configured to detect aspects of a physical environmentin proximity to the device; and one or more processors configured to:determine, via the one or more sensors, the aspects of the physicalenvironment in proximity to the device; based on determining the aspectsof the physical environment, defining one or more zones for the user tointeract with a physical peripheral input device and a virtual displayprovided by the display system; and transmit, via the transceiver, tothe display system, information indicative of the one or more zones. Theone or more sensors may include an optical, electromagnetic, orultrasound imager to detect the aspects of the physical environment. Theone or more zones can include a first zone that defines an areacharacterized in that a physical peripheral input device placed withinthe first zone is used by a user to interact with one or more virtualdisplays rendered by the display system. The first zone may bedetermined based on detecting a working surface that a physicalperipheral input device can be placed upon. The working surface mayinclude a substantially planar and horizontal surface. In some cases,the first zone can be determined based on identifying a location of aphysical peripheral input device within the physical environment, orbased on identifying one or more fiducial marks on a surface.

In certain embodiments, the one or more zones may include a second zonecharacterized in that a virtual display can be perceived by the user asbeing located within the second zone and interaction with a physicalperipheral device within the first zone can modify content observable bythe user rendered on the virtual display within the second zone. In somecases, the second zone can be selected to align with a vertical physicalboundary detected within the physical environment. The second zone maybe defined to align with a vertical physical boundary detected withinthe physical environment. The second zone can be defined based on a typeof physical peripheral device detected within the first zone. In someimplementations, the one or more zones can include a third zone, thethird zone characterized in that content displayed on a virtual displaycan be perceived by the user as being located within the third zone andwhere interaction of the user with the content modifies operation of aphysical peripheral input device located within the first zone. Thelocation of the third zone can be determined based on a location of aphysical peripheral input device being detected within the first zone. Aplurality of third zones can be determined, each corresponding to arespective peripheral input device. The location of the third zone maybe determined based on a biomechanical model of the user such that thelocation enables the user to physically reach the third zone using anappendage. In some cases, the information indicative of the one or morezones may include spatial coordinates of the one or more zones. The oneor more zones may be determined based on a location of the hub devicewithin the physical environment, or based on locations of each of aplurality of hub devices, where the hub device is included in theplurality of hub devices. The one or more zones can include a fourthzone characterized in that content displayed within the fourth zone isviewable by a plurality of users interacting within an augmented realityor virtual reality virtual reality environment; and where contentdisplayed within the second zone is viewable by only one user of theplurality of users. In some cases, the one or more sensors can includean inertial measurement unit (IMU) and the one or more processors areconfigured to determine the location of the hub device within thephysical environment via the IMU.

In some embodiments, a method of operating a hub to interact with one ormore peripheral devices within an AR/VR workstation environment caninclude detecting a first physical peripheral device within a physicalenvironment; determining a location of the first physical peripheraldevice within the physical environment; establishing a communicativecoupling with, and receiving data from, the first physical peripheraldevice and an AR/VR-based display; facilitating a transfer of receiveddata between the first physical peripheral device and the AR/VR-baseddisplay; determining an area to orient a virtual display relative to thefirst physical peripheral device within the physical environment basedon the determined location of the first physical peripheral device; andgenerating control data configured to cause the AR/VR-based display toproject the virtual display in the determined area. In certainembodiments, the control data can be further configured to cause theAR/VR-based display to maintain a spatial relationship between the firstphysical peripheral device and the projected virtual display as thefirst physical peripheral device is moved within the physicalenvironment. The method can further include receiving polling data fromone or more sensors, the polling data corresponding to physicalcharacteristics of a physical environment around the first physicalperipheral device, where determining the area to orient the virtualdisplay can be further based on the physical characteristics of thephysical environment.

In further embodiments, a movement and location of a head-mounteddisplay (HMD) of the AR/VR-based display and the hub can be trackedusing a first tracking system, and where the determining a location ofthe first physical peripheral device within the physical environment istracked via the hub using a second tracking system, the method furthercomprising: providing data corresponding to the determined location ofthe first physical peripheral device within the physical environment tothe HMD, the provided data configured to cause an integration oftracking via the second tracking system with tracking via the firsttracking system. The method can further include detecting a secondphysical peripheral device within the physical environment; determininga location of the second physical peripheral device within the physicalenvironment; establishing a communicative coupling with, and receivingdata from the second physical peripheral device; and coalescing thereceived data from the first and second physical peripheral devices intoaggregate peripheral data, where the aggregate peripheral data can betransferred to the AR/VR-based display via the hub instead of thereceived data from each of the first and second physical peripheraldevices. In some cases, the hub may be integrated with one of akeyboard, smart device, wearable device, or standalone entity.

In some embodiments, a system to interface a user with a computer caninclude a display configured to present, to a user, one or more imagesto enable a virtual reality or augmented reality virtual realityenvironment for the user; and one or more processors coupled to thedisplay, the one or more processors configured to: induce the display torender the one or more images to enable the virtual reality or augmentedreality virtual reality environment for the user; and induce the displayto render one or more layers to present, to the user, a representationof a physical interface device within the virtual reality or augmentedreality virtual reality environment, wherein the representation appearsto the user to be at a location within the virtual reality or augmentedreality virtual reality environment corresponding to a physical spatiallocation of the physical interface device. The one or more layers topresent the representation of the physical interface device may includean input element layer representing one or more input elements of thephysical interface device; an indicia layer representing one or moreindicia associated with the input elements; a feedback layer providinguser feedback of a state of the one or more input element; or a bodylayer representing a body of the physical interface device. The one ormore layers can include a proximal field layer representing a fieldproximal to the physical interface device, or a body part layerrepresenting a body part interfacing with the physical interface device.In some implementations, one of the one or more layers can berepresented as being partially transparent, where the one or morepartially transparent layers can be superposed on another one of the oneor more layers; or another one of the one or more layers is superposedon the one or more partially transparent layers. In some aspects, theone or more layers may include features representative of a depth of anobject in relation to the physical interface device.

In certain embodiments, rendering the one or more of the layers or achange in rendering the one or more layers can be in response totrigger, the trigger including one or more of: a configuration of thephysical interface device; an arrangement of the physical interfacedevice; a detection of a physical object associated with the system; acontextual event associated with the system; and a user interaction withthe system. The configuration of the physical interface device mayinclude a change in a form factor of the physical interface device. Insome aspects, the arrangement of the physical interface device caninclude a change in an orientation of the physical interface deviceand/or a location of the physical interface device. In some embodiments,the location can be a location upon a work surface; and the one or morelayers can include a proximal field layer representative of the worksurface. The physical object can be an interoperable user interfacedevice with the physical interface device; and the detection of thephysical object may include detecting a proximity of the physical objectto the physical interface device.

In further embodiments, the user interaction with the system can includeone or more of: a gesture made by the user; a user interfacing with thephysical interface device; or detection of a user's intention tointerface with the physical interface device. In some cases, the one ormore of the layers can be user customizable. The representation of thephysical interface device may include an image of the physical interfacedevice bound to a physical size of the physical interface device; andone of the one or more of the layers can extend beyond the image of thephysical interface device bound to the physical size of the physicalinterface device. The system can further including a sensor systemconfigured to sense a real-world arrangement of the physical interfacedevice, where the one or more processors can be configured to induce thedisplay to render the one or more layers based on the sensor systemsensing the real-world arrangement of the physical interface device.

In certain embodiments, a system comprising one or more processors maybe configured to track a location of an input device within a physicalenvironment via a three-dimensional (3D) tracking system, where theinput device is coupled to a virtual reality display system and whereinthe tracking the location of the location of the input device is usedfor interacting with the virtual reality display system, and modify atracking parameter of the 3D tracking system while tracking the locationof the input device based on the determined location of the input devicewithin the physical environment. A power consumption for the 3D trackingsystem may correspond to a tracking parameter of the 3D tracking system.The power consumption of the input device may be modified depending onthe tracking parameter of the 3D tracking system. In some cases, thetracking parameter can be further modified based on a biometric model ofa user of the input device where the biometric model can be used by theone or more processors to determine a location of the input devicecorresponding to the user holding the input device with an appendage(e.g., hand, fingers).

In some implementations, modifying the tracking parameter of the inputdevice may further includes tracking the location of the input deviceaccording to a first tracking accuracy profile in response todetermining that the input device is operating “in-air” in 3D space, andtracking the location of the input device according to a second trackingaccuracy profile in response to determining that the input device isoperating along a two-dimensional (2D) surface (e.g., a physicalsurface). In some embodiments, the first tracking accuracy profile canhave a lower tracking accuracy than a tracking accuracy of the secondtracking accuracy profile, and the second tracking accuracy profile ofthe 3D tracking system can be obtained based on optimizing tracking ofthe input device using the 3D tracking system for tracking along the 2Dsurface based on determining that the input device is operating alongthe 2D surface. In some cases, modifying the tracking parameter of theinput device can further include determining that the input device is ata location indicative of the input device being “in-air” in 3D space,determining that the input device is at a location indicative of a useroperating the input device “in-air” in 3D space and bracing their armagainst a surface, and tracking the location of the input deviceaccording to a third accuracy tracking profile in response todetermining that the input device is at a location indicative of theinput device being operating against a physical surface. In certainaspects, the first tracking accuracy profile can have a lower trackingaccuracy than a tracking accuracy of the second tracking accuracyprofile, where the third tracking accuracy profile has a higher trackingaccuracy than the tracking accuracy of the second tracking accuracyprofile, and where the third tracking accuracy profile has a lowertracking accuracy than the tracking accuracy of the second trackingaccuracy profile.

In some embodiments, determining that the input device is at a locationindicative of a user operating the input device “in-air” in 3D space andbracing their arm against a surface may include at least one of: usingan imaging sensor to detect an orientation of the input device or theuser; using a biomechanical model of the user to determine a locationindicative of a portion of user being in a determined position whileoperating the input device; or detecting micro-tremors of the inputdevice induced by the user while the input device is operated, wheremicro-tremors at or greater than a threshold amplitude are indicative ofthe input device operating “in-air” and the user operating the inputdevice without bracing their arm against the surface, and wheremicro-tremors below the threshold amplitude are indicative of the inputdevice operating “in-air” and the user operating the input devicebracing their arm against the surface.

Some embodiments may track the input device using an additional trackingsystem while the input device is operating along the 2D surface. In somecases, the additional tracking system includes a sensor integratedwithin the input device. The sensor may be configured to sense a changein value corresponding to a movement of the input device along the 2Dsurface. The additional tracking system can be a tracking system from alist including: a capacitance-based touch-sensitive 2D tracking system,a resistance-based touch-sensitive 2D tracking system, an ultrasoundtracking system, an electromagnetic tracking system, and an opticaltracking system. The additional tracking system can include an emitteror a sensor integrated with an additional input peripheral device or hubdevice to track the location of the input device. The input device canbe a stylus and the sensor or emitter of the additional tracking systemcan be configured to track the stylus. The input device may be any froma list including a stylus, a mobile device, a computer mouse, apresenter device, and a wearable device.

In certain embodiments, the 3D tracking system can track in three axesin a Cartesian coordinate system including a first, second, and thirdaxis, and in response to determining that the input device is operatingalong the 2D surface, the one or more processors may further be used todetermine which of the first, second, and third axes substantiallydefine a contour of the 2D surface, and suspend tracking along any ofthe first, second, and third axes that does not substantially define thecontour of the 2D surface while the input device is determined to beoperating along the 2D surface. In some cases, the one or moreprocessors can be configured to determine a distance from the inputdevice to a peripheral device, the input device and the peripheraldevice being separate and communicatively coupled to the same computersystem, and modify a tracking precision of the input device whiletracking the location of the input device based on the determineddistance from the input device to the peripheral device. In furtherembodiments, the one or more processors can be configured to modify thetracking precision of the input device according to a first trackingsensitivity profile in response to determining that the input device iswithin a threshold distance from the peripheral device, and modify thetracking precision of the input device according to a second trackingsensitivity profile in response to determining that the input device isnot within a threshold distance from the peripheral device. The firsttracking sensitivity profile can have a higher precision measurementthan a precision measurement of the second tracking sensitivity profile.

In some embodiments, the one or more processors can be configured tochange an operation of the input device based on the tracked location ofthe input device within the physical environment. The 3D tracking systemcan includes a plurality of sensor and/or emitter and where a firstsubset of the plurality of emitters and/or sensors is selected fortracking the input device based on a determination that the input deviceis at a first physical location and where a second subset of theplurality of emitters and/or sensors is selected for tracking the inputdevice based on a determination that the input device is at a secondphysical location. The second subset of the plurality of emitters and/orsensors can be selected based on a determination that the second subsetof the plurality of emitters and/or sensors is less capable of trackingthe input device at the second physical location. In some cases, thesecond subset of the plurality of emitters and/or sensors can beselected based on determining a movement vector of the input device. Themovement vector can be determined using an IMU of the input device. Insome cases, the parameter can include an accuracy, precision,sensitivity, gain, amount of hysteresis. The parameter can be one ofseveral parameters each corresponding to a dimension of 3D space trackedby the 3D tracking system and wherein each of the several parameters areindependently modified depending on the tracked location of the inputdevice. The 3D tracking system may include a plurality of trackingtechnologies and wherein the one or more processors are configured tofuse results from using the plurality of tracking technologies fortracking the location of the input device. In some embodiments, theparameter includes a confidence value or gain of at least one of theplurality of tracking technologies and wherein modifying the parametermodifies an amount that results from one of the tracking technologiescan be used for tracking the location of the input device by fusing theplurality of tracking technologies.

Modifying the tracking parameter of the 3D tracking system can be basedon determining that the input device enters a determined zone whichcorresponds to an operational mode of the input device. In some cases,the zone can be determined by an additional physical peripheral device.The additional physical peripheral device can be a keyboard, mouse,smartphone, tablet, speaker, Internet of Things (IoT) device, hubdevice, or the like. The zone can be determined based on determinedlocations of the input device as the input device can be used tointeract with a virtual reality environment enabled by the virtualreality display system, or based on a proximity of the input device withan additional physical peripheral device. In some aspects, an indicationof the zone can be rendered by the virtual reality display system toindicate to the user the location of the zone in physical space. Theinput device can be a stylus and the zone includes a 2D surface that thestylus can be operated on to manipulate one or more images rendered bythe virtual reality display system. In some embodiments, the determinedlocation that the modifying the tracking parameter is based upon can berelative to an additional physical input or hub device, or to aperceived location of a virtual display or virtual object rendered bythe virtual reality display. The virtual reality display can include ahead mounted display (HMD) and the 3D tracking system can include asensor or an emitter mounted on the HMD.

In certain embodiments, modifying the tracking parameter of the inputdevice based on the tracked location of the input device within thephysical environment can cause an operation including one of a list ofoperations including changing a function of one or more buttons on theinput device based on a contextual usage of the input device, changingan operation of the input device in response to the input device beingmoved to a same location as a virtual object or physical object,changing a visual presentation of a virtual feature of the input devicein response to the input device being moved to a same location as thevirtual object the physical object, and initiating a haptic stimulus bya haptic device coupled to the input device in response to the inputdevice being moved to a same location as the virtual the physicalobject.

BRIEF DESCRIPTION OF THE FIGURES

Aspects, features and advantages of embodiments of the presentdisclosure will become apparent from the following description ofembodiments in reference to the appended drawings in which like numeralsdenote like elements.

FIG. 1A is a block system diagram showing an embodiment of a virtualreality environment system according to the present invention.

FIG. 1B illustrates a particular embodiment of a virtual reality systemwith a sensor and tracking system.

FIG. 1C illustrates a particular embodiment of the view of a virtualreality environment through an HMD.

FIG. 1D illustrates a particular embodiment of a virtual realityenvironment with multiple virtual displays.

FIG. 1E is a high level diagram of an embodiment of an augmentation andtracking system operation.

FIG. 2 is a block diagram showing an embodiment of a networked systemfor providing content to the virtual reality environment system of FIG.1A.

FIG. 3 is a block system diagram showing an embodiment implementation ofa computer of the virtual reality environment system of FIG. 1A.

FIG. 4 is a block system diagram showing an embodiment implementation ofa user interface device of the virtual reality environment system ofFIG. 1A.

FIG. 5 is a plan view showing an embodiment of a user interface deviceof the virtual reality environment system of FIG. 1A.

FIG. 6 is a schematic diagram showing an embodiment graphicalenvironment to represent the user interface device of FIG. 5.

FIGS. 7 and 7A include a schematic diagram showing an embodimentgraphical environment to represent the user interface device of FIG. 5.

FIG. 8 is a perspective view showing an embodiment of a user interfacedevice of the virtual reality environment system of FIG. 1A.

FIG. 9 is a flow chart showing an embodiment process for interfacing auser with a computer of the virtual reality environment system of FIG.1A.

FIG. 10 is a schematic diagram showing an embodiment of a user interfacedevice of the virtual reality environment system of FIG. 1A.

FIG. 11 is a flow chart showing an embodiment process for interfacing auser with a computer of the virtual reality environment system of FIG.1A.

FIG. 12 is a plan view of an embodiment of a user interface device ofthe virtual reality environment system of FIG. 1A.

FIG. 13 is an exploded diagram showing an embodiment representation ofthe user interface device of FIG. 12 for a graphical environmentgenerated by the system of FIG. 1A.

FIG. 14 is a schematic diagram showing an embodiment graphicalenvironment to represent the user interface device of FIG. 12.

FIG. 15 is a schematic diagram showing an embodiment graphicalenvironment to represent the user interface device of FIG. 12.

FIG. 16 is a schematic diagram showing an embodiment graphicalenvironment to represent the user interface device of FIG. 12.

FIG. 17 is a flow chart showing an embodiment process for interfacing auser with a computer of the virtual reality environment system of FIG.1A.

FIG. 18 shows a simplified diagram of a peripheral-centric augmentedworkstation environment (AWE), according to certain embodiments.

FIG. 19 shows an example of an augmented workstation environment,according to certain embodiments.

FIG. 20 shows how content zones may track with respect to a referenceperipheral, according to certain embodiments.

FIG. 21 shows how content zones may collapse and expand, according tocertain embodiments.

FIG. 22 is a simplified flow chart showing aspects of a method foroperating an augmented workstation environment, according to certainembodiments.

FIG. 23 is a simplified flow chart showing aspects of a method foroperating a peripheral device (e.g., reference peripheral) in anaugmented workstation environment, according to certain embodiments.

FIG. 24 shows a simplified diagram of a hub-centric augmentedworkstation environment (AWE), according to certain embodiments

FIG. 25 is a simplified flow chart showing aspects of a method foroperating a hub to interact with one or more peripheral devices withinan AR/VR workstation environment, according to certain embodiments.

FIG. 26A is a simplified illustration of a user 2610 manipulating avirtual object 2650 displayed “in air” within a physical environment2600, according to certain embodiments.

FIG. 26B is a simplified illustration of a user 2610 manipulating avirtual object (a virtual display) 2670 displayed “in air” within aphysical environment 2600 with the user's arm 2640 braced on a surface2660, according to certain embodiments.

FIG. 26C is a simplified illustration of a user 2610 manipulating avirtual paper 2680 displayed on a surface 2660 within a physicalenvironment 2600, according to certain embodiments.

FIG. 27 is a simplified flow chart showing aspects of a method 2700 formodifying an operation of an input device based on a detected locationof the input device, according to certain embodiments.

FIG. 28 is a simplified flow chart showing aspects of a method 2800 formodifying tracking precision of an input device based on a detectedlocation of the input device relative to a peripheral device or avirtual object, according to certain embodiments.

FIG. 29 is a simplified flow chart showing aspects of a method 2900 forchanging an operation of an input device based on its tracked locationwithin a physical environment, according to certain embodiments.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.The same reference numbers may be used in different drawings to identifythe same or similar elements. In the following description, for purposesof explanation and not limitation, specific details are set forth suchas particular devices, structures, architectures, interfaces,techniques, etc. in order to provide a thorough understanding of thevarious aspects of the present disclosure. However, it will be apparentto those skilled in the art having the benefit of the present disclosurethat the various aspects of the claims may be practiced in otherexamples that depart from these specific details. In certain instances,descriptions of well-known devices, circuits, and methods are omitted soas not to obscure the description of the present disclosure withunnecessary detail.

Definitions

The present disclosure may be better understood in view of the followingexplanations:

As used herein, the terms “computer simulation” and “virtual realityenvironment” may refer to a virtual reality, augmented reality, mixedreality, or other form of visual, immersive computer-simulatedenvironment provided to a user. As used herein, the terms “virtualreality” or “VR” may include a computer-simulated environment thatreplicates an imaginary setting. A physical presence of a user in thisenvironment may be simulated by enabling the user to interact with thesetting and any objects depicted therein. Examples of VR environmentsmay include: a video game; a medical procedure simulation programincluding a surgical or physiotherapy procedure; an interactive digitalmock-up of a designed feature, including a computer aided design; aneducational simulation program, including an E-leaning simulation; orother like simulation. The simulated environment may be two orthree-dimensional. As used herein, the terms “augmented reality” or “AR”may include the use of rendered images presented in conjunction with areal-world view. Examples of AR environments may include: architecturalapplications for visualization of buildings in the real-world; medicalapplications for augmenting additional information to a user duringsurgery or therapy; gaming environments to provide a user with anaugmented simulation of the real-world prior to entering a VRenvironment. As used herein, the terms “mixed reality” or “MR” mayinclude use of virtual objects that are rendered as images inconjunction with a real-world view of an environment wherein the virtualobjects can interact with the real world environment. Embodimentsdescribed below can be implemented in AR, VR, or MR.

As used herein, the term “real-world environment” or “real-world” mayrefer to the physical world (also referred to herein as “physicalenvironment.” Hence, term “real-world arrangement” with respect to anobject (e.g. a body part or user interface device) may refer to anarrangement of the object in the real-world and may be relative to areference point. The term “arrangement” with respect to an object mayrefer to a position (location and orientation). Position can be definedin terms of a global or local coordinate system.

As used herein, the term “rendered images” or “graphical images” mayinclude images that may be generated by a computer and displayed to auser as part of a virtual reality environment. The images may bedisplayed in two or three dimensions. Displays disclosed herein canpresent images of a real-world environment by, for example, enabling theuser to directly view the real-world environment and/or present one ormore images of a real-world environment (that can be captured by acamera, for example).

As used herein, the term “head mounted display” or “HMD” may refer to adisplay to render images to a user. The HMD may include a graphicaldisplay that is supported in front of part or all of a field of view ofa user. The display can include transparent, semi-transparent ornon-transparent displays. The HMD may be part of a headset. Thegraphical display of the HMD may be controlled by a display driver,which may include circuitry as defined herein.

As used herein, the term “electrical circuitry” or “circuitry” may referto, be part of, or include one or more of the following or othersuitable hardware or software components: a processor (shared,dedicated, or group); a memory (shared, dedicated, or group), acombinational logic circuit, a passive electrical component, or aninterface. In certain embodiment, the circuitry may include one or morevirtual machines that can provide the described functionality. Incertain embodiments, the circuitry may include passive components, e.g.combinations of transistors, transformers, resistors, capacitors thatmay provide the described functionality. In certain embodiments, thecircuitry may be implemented using, or functions associated with thecircuitry may be implemented using, one or more software or firmwaremodules. In some embodiments, circuitry may include logic, at leastpartially operable in hardware. The electrical circuitry may becentralized or distributed, including being distributed on variousdevices that form part of or are in communication with the system andmay include: a networked-based computer, including a remote server; acloud-based computer, including a server system; or a peripheral device.

As used herein, the term “processor” or “host/local processor” or“processing resource” may refer to one or more units for processingincluding an application specific integrated circuit (ASIC), centralprocessing unit (CPU), graphics processing unit (GPU), programmablelogic device (PLD), microcontroller, field programmable gate array(FPGA), microprocessor, digital signal processor (DSP), or othersuitable component. A processor can be configured using machine readableinstructions stored on a memory. The processor may be centralized ordistributed, including distributed on various devices that form part ofor are in communication with the system and may include: anetworked-based computer, including a remote server; a cloud-basedcomputer, including a server system; or a peripheral device. Theprocessor may be arranged in one or more of: a peripheral device, whichmay include a user interface device and/or an HMD; a computer (e.g. apersonal computer or like device); or other device in communication witha computer system.

As used herein, the term “computer readable medium/media” may includeconventional non-transient memory, for example, random access memory(RAM), an optical media, a hard drive, a flash drive, a memory card, afloppy disk, an optical drive, and/or combinations thereof. It is to beunderstood that while one or more memories may be located in the samephysical location as the system, the one or more memories may be locatedremotely from the host system, and may communicate with the one or moreprocessor via a computer network. Additionally, when more than onememory is used, a first memory may be located in the same physicallocation as the host system and additional memories may be located in aremote physical location from the host system. The physical location(s)of the one or more memories may be varied. Additionally, one or morememories may be implemented as a “cloud memory” (i.e., one or morememory may be partially or completely based on or accessed using thenetwork).

As used herein, the term “communication resources” may refer to hardwareand/or firmware for electronic information transfer. Wirelesscommunication resources may include hardware to transmit and receivesignals by radio, and may include various protocol implementations,e.g., 802.11 standards described in the Institute of ElectronicsEngineers (IEEE), Bluetooth™, ZigBee, Z-Wave, Infra-Red (IR), RF, or thelike. Wired communication resources may include; a modulated signalpassed through a signal line, said modulation may accord to a serialprotocol such as, for example, a Universal Serial Bus (USB) protocol,serial peripheral interface (SPI), inter-integrated circuit (I2C),RS-232, RS-485, or other protocol implementations.

As used herein, the term “network” or “computer network” may include oneor more networks of any type, including a Public Land Mobile Network(PLMN), a telephone network (e.g., a Public Switched Telephone Network(PSTN) and/or a wireless network), a local area network (LAN), ametropolitan area network (MAN), a wide area network (WAN), an InternetProtocol Multimedia Subsystem (IMS) network, a private network, theInternet, an intranet, and/or another type of suitable network.

As used herein, the term “sensor system” may refer to a system operableto provide position information concerning input devices, peripherals,and other objects in a physical world that may include a body part orother object. The term “tracking system” may refer to detecting movementof such objects. The body part may include an arm, leg, torso, or subsetthereof including a hand or digit (finger or thumb). The body part mayinclude the head of a user. The sensor system may provide positioninformation from which a direction of gaze and/or field of view of auser can be determined. The object may include a peripheral deviceinteracting with the system. The sensor system may provide a real-timestream of position information. In an embodiment, an image stream can beprovided, which may represent an avatar of a user. The sensor systemand/or tracking system may include one or more of a: camera system; amagnetic field based system; capacitive sensors; radar; acoustic; othersuitable sensor configuration, optical, radio, magnetic, and inertialtechnologies, such as lighthouses, ultrasonic, IR/LEDs, SLAM tracking,light detection and ranging (LIDAR) tracking, ultra-wideband tracking,and other suitable technologies as understood to one skilled in the art.The sensor system may be arranged on one or more of: a peripheraldevice, which may include a user interface device, the HMD; a computer(e.g., a P.C., system controller or like device); other device incommunication with the system.

As used herein, the term “camera system” may refer to a systemcomprising a single instance or a plurality of cameras. The camera maycomprise one or more of: a 2D camera; a 3D camera; an infrared (IR)camera; a time of flight (ToF) camera. The camera may include acomplementary metal-oxide-semiconductor (CMOS), a charge-coupled device(CCD) image sensor, or any other form of optical sensor in use to formimages. The camera may include an IR filter, which can be used forobject tracking. The camera may include a red-green-blue (RGB) camera,which may be used for generation of real world images for augmented ormixed reality simulations. In an embodiment different frames of a singlecamera may be processed in an alternating manner, e.g., with an IRfilter and for RGB, instead of separate cameras. Images of more than onecamera may be stitched together to give a field of view equivalent tothat of the user. A camera system may be arranged on any component ofthe system. In an embodiment the camera system is arranged on a headsetor HMD, wherein a capture area of the camera system may record a fieldof view of a user. Additional cameras may be arranged elsewhere to trackother parts of a body of a user. Use of additional camera(s) to coverareas outside the immediate field of view of the user may provide thebenefit of allowing pre-rendering (or earlier initiation of othercalculations) involved with the augmented or virtual reality renditionof those areas, or body parts contained therein, which may increaseperceived performance (e.g., a more immediate response) to a user whenin the virtual reality simulation. This can be an important aspect toensure safe and pleasant use of VR. The camera system may provideinformation, which may include an image stream, to an applicationprogram, which may derive the position and orientation therefrom. Theapplication program may implement known techniques for object tracking,such as feature extraction and identification. Examples include thespeed-up robust features (SURF) algorithm. An example of a camerapresently available is the Pro C920 or C930 Full HD by Logitech.

As used herein, the term “user interface device” may include variousdevices to interface a user with a computer, examples of which include:pointing devices including those based on motion of a physical device,such as a mouse, trackball, joystick, keyboard, gamepad, steering wheel,paddle, yoke (control column for an aircraft) a directional pad,throttle quadrant, pedals, light gun, or button; pointing devices basedon touching or being in proximity to a surface, such as a stylus,touchpad or touch screen; or a 3D motion controller. The user interfacedevice may include one or more input elements. In certain embodiments,the user interface device may include devices intended to be worn by theuser. Worn may refer to the user interface device supported by the userby means other than grasping of the hands.

As used herein, the term “input element” or “user interface” may referto an object that the user interacts with to provide user input to acomputer system. The input element or user interface may include a usermanipulatable object, examples of which include: steering wheel; stick;pedal; mouse; keys; buttons; control pad; scroll wheel; flight yoke;light pen or other stylus device; glove, watch or other wearable device;rotational elements such as a dial or knob, motion detector; touchsensitive device. The user input element may be adapted to provide aninput in other manners, including by the determination of a movement ofa body part, including a touch pad.

As used herein, the term “totem” may, for example, include one or morephysical or virtual objects which are manipulatable by the user to allowinput or interaction with the AR, MR, or VR system. The totem may beused as a user interface device or a controller or as an object invirtual space. Some totems may take the form of inanimate objects, forexample a piece of metal or plastic, a wall, a surface of table.Alternatively, some totems may take the form of animate objects, forexample a hand of the user. The totems may not actually have anyphysical input structures (e.g., keys, triggers, joystick, trackball,rocker switch). Instead, the totem may simply provide a physicalsurface, and the AR, MR or VR system may render a user interface so asto appear to a user to be on one or more surfaces of the totem. Forexample, the AR, MR or VR system may render an image of a computerkeyboard or trackpad to appear to reside on one or more surfaces of atotem. For instance, the AR system may render a virtual computerkeyboard and virtual trackpad to appear on a surface of a thinrectangular plate of aluminum which serves as a totem. The rectangularplate may not itself have any physical keys or trackpad.

As used herein, the term “IMU” refers to an Inertial Measurement Unitwhich may measure movement in six Degrees of Freedom (6 DOF), along x,y, z Cartesian coordinates and rotation along 3 axes—pitch, roll andyaw. In some cases, certain implementations may utilize an IMU withmovements detected in fewer than 6 DOF (e.g., 3 DOF as further discussedbelow).

As used herein, the term “keyboard” may refer to an alphanumerickeyboard, emoji keyboard, graphics menu, or any other collection ofcharacters, symbols or graphic elements. A keyboard can be a real worldmechanical keyboard, or a touchpad keyboard such as a smart phone ortablet On Screen Keyboard (OSK). Alternately, the keyboard can be avirtual keyboard displayed in an AR/MR/VR environment.

As used herein, the term “fusion” may refer to combining differentposition-determination techniques and/or position-determinationtechniques using different coordinate systems to, for example, provide amore accurate position determination of an object. For example, datafrom an IMU and a camera tracking system, both tracking movement of thesame object, can be fused. A fusion module as describe herein performsthe fusion function using a fusion algorithm. The fusion module may alsoperform other functions, such as combining location or motion vectorsfrom two different coordinate systems or measurement points to give anoverall vector.

As used herein, the numbering of elements is consistent from figure tofigure, so that the same element is shown in different figures.

Overview

Referring to FIG. 1A, a virtual reality system 2 includes a computer 4,which can be capable of providing a virtual reality environment a user.The system 2 includes a display, which can be embodied as an HMD 8,virtual reality display (not illustrated), or other display capable ofproviding computer-rendered images to a user. The computer 4 canfacilitate display of the virtual reality environment via the HMD 8 (byincluding a display driver, for example). The system 2 includes aphysical user interface device 10 to enable a user to interface with thevirtual reality environment facilitated by the computer 4. The userinterface device 10 can receive input from the user for user control ofthe virtual reality environment. The system 2 can include a sensor andtracking system 12 to provide position and orientation information tothe computer 4, e.g. of a body part of a user, which can include a userhead orientation and/or hand orientation or of another object which mayinclude a peripheral device or other object as will be discussed.

FIG. 1B illustrates a particular embodiment of a virtual reality systemwith a sensor and tracking system, which can be similar to sensor andtracking system 12. A user 6 is illustrated as wearing HMD 8. Twoperipherals or input devices are shown—a keyboard 102 and a mouse 104.The location and movement of the keyboard 102 and/or mouse 104 (and/orhands of user 6) can be monitored with one or more cameras or othersensors. Shown in FIG. 1B are sensors 108 and 110 on keyboard 102, avideo conference camera 112, a webcam 114 and a security camera 116.Cameras can also be mounted on HMD 8, elsewhere on the user 6, on adrone, or anywhere else with a view of the physical environment. Eachcamera can be a stereo camera, with two side-by-side image sensors toprovide a 3D image with depth, or can otherwise be used to determinelocation of an object within 3D space (e.g., by using other depthsensing technologies or by determining position information by combiningimages from multiple images obtained from different viewpoints). Asshould be understood, various other sensor technologies can be used todetermine a location of a peripheral device (e.g., keyboard 102 or mouse104) within a physical environment. For example, ultrasound,electromagnetic, inertial sensors mounted to peripheral devices, opticalsensors, can be used in any combination to obtain position information.In certain embodiments, sensor fusion techniques can be used to combinepositioning information obtained from different types of sensors or fromdifferent viewpoints to determine a position.

One or more of the cameras or other sensors can detect objects in thephysical environment and can identify those objects through a variety ofmethods as described in more detail below. For example, a coffee cup 118may be detected from a barcode 120, an RFID tag, or other identifier.For an RFID tag, instead of, or in addition to, using a camera, an RFIDreader could be used. Alternately, a computer 4 can instruct thekeyboard 102 or mouse 104 to flash an indicator light so it can beidentified. The cameras or other sensors can detect which peripheral orobject the user's hands are near or touching, and the movement of theuser's hands with respect to that object. In certain embodiments,imaging sensor(s) (e.g., visual, ultrasound sensors) can be used toclassify certain objects (e.g., utilizing object detection techniques.An input device may include communication capabilities (e.g., a wirelesstransceiver) and be able to provide identification information tocomputer 4.

A space near or on any of the objects can be designated as a gesturespace. For example, a determine three-dimensional volume of airspaceabove the keyboard can be used for hand gestures which can beinterpreted as commands for computer 4 for a user to interact with avirtual reality environment. Multiple gesture spaces can be defined. Forexample, a gesture near the mouse 104 may have a different meaning, oruse different gestures, than gestures over the keyboard 102 or on thedisplay 106. More description is provided below.

As described in more detail below, an image of the object or peripheraldevice (e.g., a captured image or an image provided by a peripheraldevice) can be imported into the virtual reality environment forpresentation to a user interacting within the virtual realityenvironment. For example, an image of keyboard 102 can be imported intothe virtual reality environment for a user to interact with.Alternately, the image can be augmented or otherwise changed or moved.For example, the types of keys shown on a keyboard representation couldbe changed from alphanumeric to emojis. An input that isn't present canbe projected onto the object in the virtual reality environment, such asa keypad on top of mouse 104 or even on coffee cup 118.

FIG. 1C illustrates a particular embodiment of the view 122 of a virtualreality environment through HMD 8. In this embodiment, the virtualreality environment is used to display multiple windows of a computermonitor without requiring multiple physical monitors. As shown, virtualdisplays 124, 126 and 128 are shown. Also shown is a virtualrepresentation 130 of the physical keyboard and user's hands, injectedinto the virtual reality environment.

FIG. 1D illustrates a particular embodiment of a virtual realityenvironment with multiple virtual displays, such as display 132. Inaddition, a virtual display 134 of a physical keyboard and/or the usercan be provided. Finally, virtual representations of other physicalobjects can be provided, such as a virtual coffee cup 136 and a virtuallamp 138. This enables the user, for example, to grab the real coffeecup and take a drink while wearing the HMD 8. A notepad and pen areother objects that can be identified and injected into the virtualreality environment, so a user can physically write notes while wearingHMD 8. Other items, such as a user's watch, can be injected so the usercan check the time on their real watch. An office door behind the usercan be monitored, and can be displayed on a rear-view mirror in thevirtual reality environment.

FIG. 1E is a high level diagram of an augmentation and tracking systemoperation. A camera 142 can provide video to an augmentation engine 150and a tracking engine 152. Video from other cameras, and/or othersensing data from other sensors can be provided. The video and othersensing data can be used for two purposes. First, the video and/orsensor data is used to provide a realistic (e.g., “real worldrepresentation) display of one or more peripherals and the user's handsin the virtual reality environment. Second, the video and/or sensor datais used to track the user movements with respect to the peripherals.

Separately, control inputs from keyboard 148 can be provided toaugmentation engine 148 and tracking engine 152. Determination of whichkey is depressed, or about to be depressed, can be detected by an actualkey press which can generate a corresponding signal, a proximity sensoron the key, and/or from the camera 142 or other sensors. The controlsignals can be provided to “press” the appropriate key in the virtualreality environment, as well as determine which key should be augmented.The video and control signals, as augmented and modified, can beprovided to application and operating system software 154 for the HMD 8.

In one embodiment, a representation 146 of the keyboard and user's handsis displayed on a physical monitor 144, and then the display of thephysical monitor is imported into the HMD 8. Alternately, representation146 is only generated in the HMD, providing more flexibility onpositioning, such as under a desired one of multiple virtual monitors inthe virtual reality environment.

Network and Computer Elements

FIG. 2 is a block diagram showing an embodiment of a networked system 22for providing content to the virtual reality system of FIG. 1A. Acontent server 14 may provide some or all of the video content or otherdata or controls to the computer 4 of FIG. 1A, through a network 20 anda receiver 16 in communication with computer 4. Network 20 can be theInternet, a Wide Area Network (WAN), a Local Area Network (LAN) or anyother network or networks or communication path. Alternately, thecomputer may generate and/or control the virtual reality environmentfrom locally stored instructions. In one embodiment the computer 4includes a memory with instructions executable for generation of thevirtual reality environment. The number of content servers 14, receivers16, and networks 20 can be modified as appropriate for a particularimplementation, such as where content is provided from multiple contentservers, or computers of other users in a shared video game. Thereceiver 16 may connect to the network 20 via wired and/or wirelessconnections, and thereby communicate or become coupled with contentserver 14, either directly or indirectly. Alternatively, receiver 16 maybe associated with content server 14 through any suitable tangiblecomputer-readable media or data storage device, data stream, file, orcommunication channel.

FIG. 3 is a block diagram of computer 4. Computer 4 may be implementedwith a processor 26 (sometimes referred to herein as a “hostprocessor”). The computer 4 can include memory such as main memory 28and secondary memory 30. Main memory 28 may include a random-accessmemory (RAM) or another type of dynamic storage device that storesinformation and instructions for execution by processor 26. Secondarymemory 30 may include hard drives or other memory devices that may storeinstructions or data that may not be directly accessible by processor26. The computer 4 can include an input/output (I/O) system 32 tointerface the computer 4 with peripheral and input devices, includingthe HMD 8 and user interface device 10 of FIG. 1A. One or more devicedrivers (not shown) may facilitate communication with the peripheraldevices. The I/O system 32 may include a display (not shown) in additionto the HMD 8, which may display images of the virtual realityenvironment application, operating system application or other imagesassociated with the virtual reality environment. The display may allowconfiguration of system 2 by the user without needing to wear the HMD 8,or configuration of system 2 by any other person, or to allow asecondary non-VR display of the virtual reality environment, e.g. forobservational, run-time configurational, safety or any other purpose.The display may be a holographic display capable of renderingthree-dimensional images for a user without the need of or bysupplemented HMD 8. The I/O system 32 may include an audio output device(not shown) such as a speaker, which is preferably coupled to hostprocessor 26 via associated audio circuitry such as, but not limited toamplifiers, filters, and other circuitry known to the skilled person toprovide sound output to the user when an audio event occurs during theimplementation of an application program. In embodiments the audiooutput device and associated circuitry may be located on the HMD 8.

The computer 4 may perform operations based on instructions that may beread into the main memory 28 from a computer-readable medium or fromanother device via communication resources. The instructions containedin the main memory 28 can cause processor 26 to perform processes thatwill be described later. Alternatively, hardwired circuitry may be usedin place of or in combination with instructions to implement saidprocesses. Thus, various implementations are not limited to any specificcombination of hardware circuitry and software.

The computer 4 may implement instructions for determining from an imagestream provided by the camera system the arrangement of objects in thecapture area. For object tracking the instructions may implement knowntechniques, such as feature extraction and identification. Examplesinclude the speed-up robust features (SURF) algorithm.

Example implementations of the computer 4 include: a personal computer(PC); workstation; laptop; a home video game console system, which istypically connected to a television set or other display; a set-top box,which can be used, for example, to provide interactive televisionfunctions to users; or a network or internet-computer; a media player(such as an MP3 player); a subnotebook/netbook; a tablet computer; asmartphone; a cellular telephone; a personal digital assistant (PDA);other similar electronic device; other suitable device. In certainembodiments, the computer 4 may be incorporated as part of the HMD 8and/or user interface device 10. The computer 4 may be operable undervarious operating systems.

FIG. 4 is a block diagram of user interface device 10. The userinterface device 10 can include a local processor 34. By local it ismeant that the local processor 34 is separate from the host processor 26of the computer 4. The local processor 34 may be provided withinstructions to wait for commands or requests from the computer 4 decodethe command or request, handle/control input and output signalsaccording to the command or request, control sensors of user interfacedevice 10 for imaging a real world environment in proximity to interfacedevice 10, communicate with other user interface devices, and/orsupplement host processor 26 for generation of and interaction within avirtual reality environment. In addition, the local processor 34 andhost processor 26 may operate independently of the computer 4 by readingsensor signals and calculating appropriate forces from those sensorsignals, time signals, and stored or relayed instructions selected inaccordance with a host command. Local memory 36, such as RAM and/or ROM,is coupled to local processor 34. Processor 26 is similarly coupled toits own local memory 28, 30.

The user interface device 10 includes an I/O system 38 to provideinformation to/from the local processor 34. The I/O system 38 may varydepending on the configuration of the user interface device 10. Inembodiments the I/O system 38 may be incorporated, connected to, or incommunication with the sensor system described below. Said inputelements may include an associated sensor system coupled to the I/Osystem 38, which may include one or more of the following: opticalsensor systems; optical encoders; potentiometers; velocity sensors;acceleration sensors; strain gauge; resistive sensors; capacitive (e.g.touch) sensors, or any other type of sensors. A sensor driver (notshown) may be implemented to convert sensor signals to signals that canbe interpreted by the local processor 34 or host processor 26.

The user interface device 10 is coupled to the computer 4 bycommunication resources 40, an example of which includes abi-directional bus such as an RS232 serial interface, RS-422, RS-485,12C, Universal Serial Bus (USB), MIDI, Firewire or other protocols knownto the skilled person or a parallel bus or wireless link, such as BTIeor proprietary wireless protocols, for example. In certain embodiments,the communication resources 40 can supply power from the computer 4 tothe user interface device 10 (sometimes simply referred to as an“interface device”). In certain embodiments, a dedicated power supplycan be implemented to power user interface device.

The user interface device 10 may not be limited to providing an input tothe computer 4, in certain embodiments, the user interfaced deviceprovides output to the user, for example in the form of visual or and/orhaptic feedback.

Augmented and Mixed Reality

Various input devices, objects, user body parts and even air spaces maybe imported into the virtual reality environment as realistically aspossible, or may be augmented/changed. One example is highlighting akeyboard key as it is pressed or approached by the finger of a user 6 asshown in FIG. 6. Another example is changing a portion of thealphanumeric keys into emojis, as shown in FIG. 15. The changes can becontextual, dependent on user actions, the type of application beinginteracted with in the virtual reality environment, etc.

In certain embodiments, a physical appearance of aspects of user inputdevices or other objects can be augmented or changed. A particularkeyboard key, knob, mouse button, roller, etc. can be highlighted bylighting, changing color, changing size, making it move or wiggle,changing its shape, etc. For example, a user's hands can have the colorchanged to improve the contrast with a keyboard or other device. Theuser's hands can be made partially transparent to allow seeing the keysotherwise obscured by the user's hands. The user's hands can be shownwith gloves or jewelry added, for example. The changes can becontextual, such as adding leather gloves when a user is interactingwith a steering wheel in a car racing video game, or with a joystick ina flight simulator. The user's hands could be made to appear wrinkledwith warts upon being hit with a spell in a witchcraft game.Alternately, the user can dictate and configure the changes, with theuser changes optionally being dependent on context. For example, theuser can specify a Star Trek uniform shirt sleeve on the depiction ofthe user's wrists, but only during a space exploration game or genre ofgames. Typing on Halloween can give the user an option to selectdifferent types of alternative images like “monster hands.”

In various embodiment, the augmentation can perform a variety offunctions, such as: enlarge the key text for optimum viewing; change thetransparency of your hands so you can see all the keyboard; change thekey layout and text to the appropriate language; change the key layoutfor more symbols and rich text; change the layout to be a fullcontextual Emoji layout; change the layout to only show your relevantgaming keys; change the colour/style of your keyboard and desk; writevirtual notes and paper and move them into the virtual space; gesturewith your hands and interact with the virtual environment; create hotkeys on different locations on the keyboard (and desk).

Augmentation can be done for productivity, music creation, video editingetc. The user input device can control the AR/VR desktop (keyboard/mouseand interactive table area) or the user input device can integrate withthird party elements such as Twitch.

Inert objects can become input devices in certain embodiments. Forexample, the virtual keyboard 11 of FIG. 6 could, in reality, be aportion of a desktop. The user can simply be moving his/her fingers andtapping on the desktop, while having the location and movement of theuser's fingers superimposed over a virtual keyboard in the virtualreality environment. The location on the desk to be used as a keyboardcan be selected by computer 4 or by the user. A user command can selectthe area using a typed command, response to a prompt, a gesture, averbal command, etc. The command can be in combination with the userpointing to the area or making a gesture over the area. The designatedarea can move in response to designated user gestures or hand positions.

In another example, a coffee mug can become a joystick or puck. Buttonscan be superimposed on the virtual representation. The buttons can beplaced in fixed locations, or can be place below where the user'sfingers happen to be while grasping the coffee cup.

In addition to augmenting or changing buttons, keys, rollers and otherinput or sensing elements, other aspects of a user interface device orother object can be changed. A color of the frame of a keyboard can bechanged, can be made to glow, flash, etc. The shape and appearance canbe changed. Typed letters/words can be presented at the top of thekeyboard, as if the keyboard had a small display at the top. A mouse canbe changed to appear like a live mouse with whiskers in the user'shands, with the whiskers and a tail moving in response to certain inputsor contextually based on what is occurring in the virtual realityenvironment. Haptic feedback to the mouse can be provided so the userfeels the mouse wriggling. The mouse might change into other objects,such as a sword, a bomb, a gun or other object depending on the contextof a game, such as a combat game. The change can be triggered by user orother actions in the virtual reality environment, such as the useraccumulating a certain number of points.

An injected user interface device video can be customized. In oneembodiment, the injection of the user interface device and user hands isaccomplished by a peripheral system providing a video to be displayed ina designated portion of the display are of the HMD. The peripheralsystem can add any desirable content in that video area, such as helpprompts on functional capabilities of the user interface device, or evenadvertisements.

In certain embodiments, camera images can be used to locate a user inputdevice or peripheral, but stored images can be used instead or toenhance the image or replace portions not visible due to being obscuredby a user's hands or another object on a desktop. The particular storedimage can be obtained by detecting or inputting the model number of theinput device.

Sensor and Tracking System

Objects to be sensed and tracked can be identified in a number of ways.User input devices can be detected upon connection to computer 4, byeither a wired or wireless connection. Image recognition can be usedwith one or more cameras to locate the user input devices. The userinput devices could also be identified by having computer 4 cause anindicator light on the device to flash, to be detected by the camera. Orthe user could be prompted to touch or manipulate the device. Imagerecognition can also be used to detect input devices. For example, alimited number of input device models may be available for use invirtual reality, and a database can store images of those models fromdifferent angles for image recognition.

In addition to connected devices, other object can be identified andinjected into the virtual reality environment. A user can put barcodelabels or RFID tags or other identifiers on the objects to beincorporated. A barcode or RFID reader can be incorporated into a userinput device, a peripheral device, a camera, the HMD, or any otherobject. Alternately, the user can use gestures, such as pointing, toindicate an object to be sensed. Alternately, a particular peripheral,such as a stylus, can be used to touch or point to desired objects. Thestylus could light up, or send out IR, radio, NFC or other signals toindicate its position. The gesture could be in combination with anaudible or typed command or other input. In this manner, the user candecide what is important to incorporate into the virtual reality space.For example, the furniture may be incorporated so the user can walkaround and see the furniture without bumping into it. Drinks or foodcould be identified so the user can eat or drink while wearing the HMD.A brush or comb could be identified, or nail clippers, so the user cando personal grooming while in the virtual reality environment, perhapsas part of a grooming instruction program.

Identified devices and objects can be injected into the virtual realityenvironment based on various triggers (described in more detail below).For example, a desk or keyboard may appear only when the userapproaches. A desk or other object could appear to avoid the userbumping into it, while other devices or objects could appear upon atrigger indicating a user need for the device or object. The triggercould be a user gesture, command, placing a device upon a surface, oreven detecting a user gazing at a device or icon.

Identified user input devices and other peripherals and objects thathave been identified can have their images captured by one or moresensors so that the images are available for injection into the virtualreality environment. The same or different sensors can also track themovement of the user input devices and other peripherals and objects, aswell as a user's hands and potentially other body parts. Differenttechnologies can be used, such as visible spectrum and IR cameras orsensors for providing images, and lasers for detecting movement. Varioussensors and user interface devices can be networked, to work incooperation for identification, locating, and tracking movement. Forexample, a keyboard can identify a particular key that was pressed, anda laser and camera can both detect a finger moving downward above thatkey. The various actions can be correlated, used to confirm, or usedwhere one of the sensors is blocked or otherwise does not have a highconfidence detection.

A gesture space can be defined in certain embodiments. For example, aspace above a keyboard and/or mouse can be defined as a gesture space.Alternately, any other space or surface can be designated for gestures.For example, a portion of the desktop, or a mousepad on the desktop, canbe turned into a touch sensor. Multiple gesture spaces or surfaces canbe used in combination, or separate gesture spaces or surfaces could beused for different purposes, based on location and context of what ishappening in the virtual reality environment.

A gesture space or surface can remain inactive until activated, so auser doesn't inadvertently cause something to happen with a gesture.Activation can be by a gesture, such as pointing to the space orsurface, by a button press, verbal or typed command, or any other input.Gesture spaces can be activated automatically based upon context, suchas activating a touchpad when a particular virtual display is presentedin the virtual reality environment. The activation can be combined withthe display of a virtual touchpad in the appropriate space in the HMD. Aphantom, faded, or other representation of the gesture space or surfacecan be shown when inactive, so the user knows where it is. For example,a square or round fog cloud could be shown in the appropriate space.

A sensor based on radar technology can be used to pick up gesturesaround a peripheral. These gestures can enable the user to performrecurrent actions that normally require pressing a non-intuitive key onthe keyboard. Some examples are set forth below.

Swipes to the left or right in within a gesture space could be used to(1) switch a window rendered within a virtual desktop, (2)enable/disable the display of a virtual keyboard, (3) based on thecontext, move forward/backward in a video, next/previous song,next/previous photo in the slideshow, etc.

If unable to quickly identify a key or special character, give theability to the user to draw it intuitively near the keyboard to inputthe desired character. This could use character prediction to reinforcethe confidence level of the gesture detected.

A sensor based on IR lasers to identify fingers on an edge or theperipheral, or all around the peripheral could be used. These laserswould be close to the table top surface and detect up to 10 fingers andtheir respective locations. The same types of use-cases can arise fromthis type of sensor as for the radar sensor. This sensor could, however,be able to pick up other types of gestures and create new use-cases,such as: detect a two or three fingers gesture that describes an actionthat aims at increasing/reducing the size of an object. This can allow auser to perform an action that complements the usage of (1) a pointingdevice, (2) voice interactions, (3) gaze interactions, by providing amulti-modal way to perform the given action. E.g. using a stylus, usingthe gesture to modify a size of the brush used while drawing, etc.

In certain embodiments, a sensor and tracking system can be incorporatedin one or more user input devices or peripherals. Alternately, they canhave portions implemented in different devices, such as object detectionsensors in a puck on the desktop, motion tracking in cameras such as awebcam, and tracking software for analysing the sensor data in thekeyboard or in computer 4, or a combination. A “smart puck” or “hub”peripheral can be provided on the desk that is able to sense and locateother peripherals (i.e., the keyboard) and track them. Depending onwhere the puck is set, a context can be assigned to it, hence enablingit to become your desk, your TV setup or another environment. The puckcan be a portable or wired device that is equipped with sensors thatallow it to perceive peripherals nearby. Such sensors can be based on IRphotodiodes (Vive-like), ultrasonic, magnetic, optical, etc.technologies.

In one embodiment, the distorted representation of the user's peripheraland hands from one camera or sensor can be rectified by applying aperspective distortion algorithm based on the computed difference ofperspective from two types of sensors.

Triggering

The appearance of a user interface device, peripheral or other objectcan be triggered by context. Also, the position and presentation can bevaried based on the same or different triggers. For example, a keyboardcan be shown in its actual position, or a fixed position relative to theactual position. The keyboard could be attached to the bottom of adisplay where typing is occurring in the virtual reality environment.Alternately, the user's gaze can be tracked, and the keyboard can followwhere the user is gazing. The triggers can be customized by the user,and can vary depending on the software application or mode of operation.

The type of user input device can be trigged by context, as well as theposition in the virtual reality environment, and when to inject and whento remove. For a message window, a keyboard can be injected below themessage window when a message is received, or the user opens themessaging application. If the user selects a fighter pilot game, ajoystick may be automatically injected. For selection of a racing game,a steering wheel can be automatically injected. The injected objectsalso won't always be put in front of the user. Thus, there is amultiple-faceted decision tree, such as: 1) trigger event detected; 2)determine real world arrangement; 3) determine, based on the context,where to inject the peripheral; 4) guide the user to the peripheral(perhaps lighting keys up or giving directional arrows or haptics orsounds or another prompt.

The appearance of particular user interface devices, peripherals orobjects can be triggered by the user approaching them, an application inthe virtual reality space requiring them at a particular point in time,an explicit user command, or a combination. The location for injectionand the type of injection is also triggered by the context, ascustomized by a user.

Inserted Graphical Environment

FIG. 6 shows a graphical environment 42 for insertion into the virtualreality environment of HMD 8. Graphical environment 42 can include videoimages shown (of the keyboard and hand), for insertion into the virtualreality environment. Graphical environment 42 can also include controlor input signals corresponding to key presses and other inputs fromkeyboard 11. Graphical environment 42 may be created by one or both ofprocessor 26 in computer 4 and processor 34 in user interface device 10.The processor(s) provide instructions for generation of a video displayor other image insertion into graphical environment 42, which isinserted into, or combined with, a virtual or augmented or mixed realityenvironment of HMD 8. Whilst reference is made to the processor 26 ofthe computer 4, it will be understood that any processing resourcearranged as part of the system 2 (various examples of which aredisclosed here in) or other processing resource in communication withthe system may be utilised. The instructions may be provided to the HMD8 for display of the virtual reality environment (including a displaydriver coupled thereto).

Embodiment arrangements of the user interface device 10, implemented asthe keyboard 11 will now be provided. Both reference numbers 10 and 11are used to emphasize that the shown keyboard 11 could be any other userinput device 10. It will be understood that the arrangements may beapplied to other implementations of the user interface device 10,various example implementations are provided herein.

Referring to FIG. 5, a keyboard 11 includes input elements 44 (keyboardkeys, touchpad, roller, mini-joystick, etc.). The sensor system 12 isarranged to provide information to help determine a real-worldarrangement of a body part of a user 6 (e.g. the hands of a userincluding digits). The real-world arrangement may be determined relativeto the user interface device 10. The real-world arrangement body part ofuser 6 may be determined during interfacing of the body part of user 6with the user interface device 10. As used herein “interfacing” mayrefer to a body part of user 6 interacting with the one or more inputelements 44 or in operative proximity to interface with said elements. Aportion or all of the sensor system 12 is operatively connected to orimbedded in the user interface device 10 in one embodiment. Inembodiments operatively connected may refer to one or more of thefollowing: the sensor system 12 physically connected to the userinterface device 10 (e.g. arranged on a body thereof); the sensor system12 physically connected to a device that the user interface device 10can be connected to (e.g. a docking station), supported by (e.g. asupport platform), or otherwise proximally aligned in relation thereto.Referring back to FIG. 5, the sensor system 12 can be arranged above thekeyboard 11 to capture interaction with the keyboard during use.

Referring to FIG. 8, peripheral device 17 is shown. The peripheraldevice 17 can include a support surface 15 for support of the keyboard11. The peripheral device includes an arm 19 extending from the supportsurface 15. A camera 13 is arranged on the arm in a position along aproximal gaze direction of user 6 to capture a proximal point of gaze ofthe user of the interface device. With such an arrangement there may beminimal mapping of the captured image of the body part and/or keyboard11 to a representative view as will be discussed.

Processor 26 and/or processor 34 can be used to determine a real-worldarrangement of the body part of user 6 based on the information of thesensor system 12 in certain embodiments. The real-world arrangement ofthe body part of user 6 may be determined entirely by the portion of thesensor system 12 operatively connected to the user interface device 10,implemented as a keyboard 11 operatively or by said portion incombination with another portion of sensor system 12 arranged on asupport portion 15 or elsewhere, examples of which include anotherperipheral device, e.g., the HMD 8, another display associated with thecomputer 4, a workstation surface (e.g. a desk) arranged device thatincludes a dedicated camera. In some cases, the sensor for detecting auser's hands can be integrated in a peripheral device, hub, an HMD andcan be used in any combination, as further described below.

Referring to FIG. 6, in embodiments the processor 26 and/or 34 caninclude instructions for creating a representative arrangement in thegraphical environment 42 of the body part of user 6 or an peripheralinput device based on the determined real-world arrangement of the bodypart of user 6 or device. The processor 26 and/or 34 can provide arepresentative arrangement of the user interface device 10 relative tothe body part of user 6. As used herein the term “representativearrangement in the graphical environment” means that the arrangement isin some way representative of the real-world arrangement, including oneor more of the position in the user field of view; orientation; scale.The representative arrangement can include the user interface device 10arranged as being aligned at the top of the graphical environment 42with the same orientation as in the real-world. The user interfacedevice can be arranged elsewhere in the graphical environment 42. Therepresentative arrangement can include the body part of user 6 arrangedwith the same physical orientation as in the real-world and arrangedwith respect to the user interface device 10 as in the real-world. Auser may be able to observe their interactions with the user interfacedevice 10 in the graphical environment 42. The user interface device 10and/or body part can be omitted in certain embodiments. In certainembodiments, either the user interface 10 or the body part of user 6 canbe omitted.

Referring to FIG. 7, the processor 26 and/or 34 and associated memorycan include instructions for displaying the body part of user 6 and userinterface 10 arranged an equivalent position in the field of view in thegraphical environment 42 to that in the field of view in the real-worldenvironment. As discussed previously, to facilitate the associatedprocessing, the sensor system 12 may provide position information fromwhich a direction of gaze and/or field of view of a user can bedetermined. Said gaze direction and field of view may thus be determinedby the processor 26 and/or 34 and the position in the real-world of thebody part of user 6 and user interface 10 determined relative thereto. Auser may have enhanced interaction with the user interface device 10 inthe graphical environment 42.

In certain embodiments, one or more cameras of the sensor system 12 canprovide images of the body part of user 6 and/or user interface device10 for rendering in the graphical environment 42. Said rendering maycomprise a simplified image for the graphical environment 42, e.g. arepresentative skin tone is implemented represent the body part, whichmay have the same outline as in the real-word, or an outline image. Saidrendering may comprise an alternative image for the graphicalenvironment 42, e.g. hands of a user are displayed as wearing gloves fora virtual reality environment including a vehicle simulator. In certainembodiments, the body part of user 6 is alternatively represented asincluding a cursor or other pointing means.

In certain embodiments, sensors of the sensor system 12 operativelyconnected to the user interface device 10 include capacitive sensorsarranged to determine the proximity of the body part of user 6 to theuser interface device 10, e.g. hovering over the keyboard. Thecapacitive sensors may be arranged on a body of the user interfacedevice 10. Sensors of the sensor system 12 operatively connected to theuser interface device 10 can include one or more cameras arranged on abody of the user interface device 10. Sensors of the sensor system 12operatively connected to the user interface device 10 can be arranged inone or more of the following manners: incorporated within the userinterface device 10, proximal a side, front, top, bottom of the userinterface device 10 (wherein a top represents the platform of the deviceand an opposed bottom represents a base); on a body of the userinterface device 10, e.g. to capture an image of an underside of a handof a user; on a user manipulatable arm extending from the user interfacedevice 10 or another device or object; or other physically connectedarrangements.

The processor 26 and/or 34 may determine the real-world arrangement ofthe body part of user 6 based on the information of the sensor system12. In certain embodiments, the real-world arrangement of the body partof user 6 may be determined entirely by the portion of the sensor system12 operatively connected to the user interface device 10. In certainembodiments, the real-world arrangement of the body part of user 6 maybe determined by a portion of sensor system 12 in combination withanother portion of the sensor system 12 arranged elsewhere on the system2, examples of which include another peripheral device, e.g. the HMD 8,another display associated with the computer 4, a work surface (e.g. adesk) arranged device that includes a dedicated camera, or another likeconfigured user interface device 10.

In certain embodiments, a processor may map the determined real-worldarrangement of a body part and/or user interface device 10 to arepresentative arrangement by means of a real-time operation, e.g. theapplication of one or more of a: rotation; translation;enlargement/reduction. The real-time operation may be user configuredand/or based on a position of the user interface device 10 relative thesensors system 12 (e.g. with reference to the embodiment associated withFIG. 8 the positioning of the keyboard relative the support portion). Asan example, referring to the embodiment associated with FIG. 5, thereal-world arrangement can be mapped through applying a real-time anglemapping to provide a point of view proximal an estimated user's gaze. Incertain embodiments, a user's gaze may be determined (by means discussedpreviously) and the real-world arrangement actively mapped to said gaze.

Referring to FIG. 6, the keyboard 11 can be aligned at the top right ofthe graphical environment 42. In certain embodiments, the keyboard 11can be alternatively aligned, including various screen edge positions orwith predetermined offsets therefrom. The body part 6 can be arrangedwith the same orientation and position with respect to the keyboard 11as in the real-world. In spite of the arrangement of the keyboard 11, auser may be able to observe their interactions with the keyboard 11 inthe graphical environment 42.

Referring to FIGS. 7 and 7A, in certain embodiments, the keyboard 11 isinjected and arranged based on a contextual event, wherein a promptwindow appears and the keyboard 11 is bottom edge aligned. For example,when a prompt window appears requiring text input, a peripheral, such asa keyboard, may be injected and aligned with the window. Alignment canoccur in a number of ways, such as snapping corners to corners,centering and parallel matching, or user customizable by altering theangle of which the keyboard is displayed. In certain embodiments, thekeyboard 11 is alternatively aligned to the promo window, includingvarious screen edge positions or with predetermined offsets therefrom.In embodiments, the contextual event may include an event in the virtualreality environment that requires user input from the user interfacedevice 10 or an electronic notification external the virtual realityenvironment (e.g. the receipt of a message).

The arrangement of the user interface device 10 can be user configured,e.g. via accessing a preferences setting of the virtual realityenvironment or other related application program. The user interfacedevice 10 may be user displaceable, e.g. by drag and drop. The userinterface device 10 can be locked in place so it doesn't move when theuser isn't actively using the prompt window.

The user interface device 10 can include a positioning system (notshown) for determination of the arrangement of the user interface device10 by the sensor system 12. The sensor system 12 can provide informationto the processor 26 and/or 34 for said determination. This informationcan come from the sensor system 12, a tracking system, the HMD 8, thecomputer 4, or from anything that may detect or see the peripheral. Thepositioning system may include one or more of the following: emitters toemit a signal including one or more of optical (including infra-red),radio waves, acoustic, magnetic field, radio; ultrasound; acharacteristic pattern; a characteristic object (including a logo). Thepositioning system may facilitate convenient and accurate recognition ofthe interface device 10 and its arrangement.

Referring to FIG. 9, a, process of interfacing a user with a computer,which may be implemented by the processor, 26 and/or 24 may include, atblock 48, obtaining, from the sensor system 12 operatively connected toa user interface 10, information to determine a real-world arrangementof a body part of user 6 interfacing with said user interface device 10.The process may include, at block 50, determining, based on saidinformation, the real-world arrangement of the body part 6 relative theuser interface device. The process may include, at block 52, generatinginstructions to display a representative arrangement of the body part ofuser 6 and user interface device based on the determined real-worldarrangement of the body part of user 6.

A sensor system 12 operatively connected to the user interface device 10may provide information to determine a physical arrangement of the bodypart of user 6 and user interface device 10 of accuracy sufficient fordetermining usage of the user interface device 10, e.g. fingersinteracting with the one or more input element 44.

Master Peripheral

In certain embodiments, a particular user interface device 10 (or otherdevice) can act as a master peripheral, hub, or anchor peripheral. Itmay include all or part of the sensor system 12 and do all or part ofthe processing of sensor data to determine the location of itself orother objects and track them. The anchor peripheral (e.g., a keyboard)can determine and optionally inject into the virtual reality environmentother nearby peripherals when contextually relevant. Additionalinteractions can be provided based upon the combination of peripherals.For example, a first type of interaction can happen when a mouse andkeyboard are brought in, and a second type when a keyboard and speakerare brought in, etc.

Referring to FIG. 10, a peripheral device 54 can include a sensor system12 arranged to provide information to determine, relative the peripheraldevice 54, a real-world arrangement of one or more objects 56 (e.g.,another user interface device, peripheral, totem, every day object,user's hand for a gesture, etc.). The sensor system 12 may beoperatively connected to the peripheral device 54. Operatively connectedmay include: incorporated within peripheral device 54; the sensor system12 physically connected to the peripheral device 54; the sensor system12 physically connected to a member that the peripheral device 54 can beconnected to, supported by or otherwise proximally aligned in relationthereto. The sensor system 12 may be operable to detect a positionperipheral device 54 within a real world environment. For example,sensor system 12 can include an IMU, imaging, or other sensors.

The object 56 may be located in a proximal field of the peripheraldevice 54. The term “proximal field” as used herein may include any twoor three dimensional space. Examples include: a work surface supportingsaid peripheral device 54; a three-dimensional space proximal userinterface device as a game controller which, in use, is held by a user;a mat or pad for use with a user interface device 10 as a pointingdevice, including a mouse or a stylus.

The processors 26 and/or 34 to determine the real-world arrangement ofthe one or more object 56 based on the information of the sensor system.The real-world arrangement of the one or more object 56 may bedetermined entirely by the portion of the sensor system 12 operativelyconnected to the peripheral device 54 or by said portion in combinationwith another portion of sensor system 12 arranged elsewhere on thesystem, examples of which include another peripheral device, e.g. theHMD 8 another display associated with the computer 4, a work surface(e.g. a desk) arranged device that includes a camera.

The processor 26 and/or 34 compiles the sensor data collected from thevarious systems to inject into the virtual reality environment arepresentative arrangement of the peripheral device 54 and the objectsarranged relative the peripheral device based on the determine thereal-world arrangement. In one embodiment, the information provided bythe sensor system 12 of the peripheral device 54 is thus used forgenerating the representative arrangement of the one or more object 56,which is then associated with the peripheral device 54.

In certain embodiments, processor 26 and/or 34 maps the arrangement ofthe one or more object 56 with the peripheral device 54 as theperipheral device is determined to move relative a reference point (e.g.another input device or sensor or other reference position) based on thedetermined real-world arrangement of the one or more object 56 relativeto the peripheral device 54.

In certain embodiments the sensor system 12 includes a camera and/orcapacitive sensors arranged on the peripheral device 54. The peripheraldevice 54 can be a dedicated body for the sensor system 12, e.g. a bodyto abut, in use, a workstation surface. The object 56 can include anyobject than may be arranged in the proximal field: other peripheraldevices including another user interface device 10; other objectsrelevant to the user environment, e.g. a beverage or foodstuffcontainer, smart phone, lamp etc. The object 56 can include thepreviously described positioning system (not shown) for determination ofthe arrangement of the user interface device 10 by the sensor system 12.

In certain embodiments, one or more cameras of the sensor system 12 ofthe peripheral device 54 provide images of the or each object 56 forrendering in the graphical environment 42. Said rendering may comprise asimplified image for the graphical environment, e.g. a representativetone is implemented represent the object. Said rendering may comprise analternative image for the graphical environment.

Referring to FIG. 11, a process of interfacing a user with a computer,which may be implemented by processor, may include, at block 58,obtaining sensor data from the sensor system 12 of a peripheral device54. The process may include, at block 60, determining, based on saidsensor data, the real-world arrangement of the one or more objects 56.The process may include, at block 62, generating instructions to injectinto the virtual reality environment a representative arrangement of theperipheral device 54 and the one or more objects 56 arranged relativethe peripheral device based on the determined real-world arrangement.

Sensors of the peripheral device 54 may provide information to determinea physical arrangement of the one or more objects 56 to a high level ofaccuracy. Other sensors (e.g., mounted to an HMD or other displayassociated with the computer 4) may, in isolation, not be capable ofproviding information with such accuracy. Moreover, using informationfrom the sensor system 12 of the peripheral device 54 to determine thereal-world arrangement of the one or more objects 56 may enable lowerprocessing resources than said alternatively arranged sensor systems 12.The peripheral device 54 may relay position information (or the positioninformation processed as said physical arrangement) of a plurality ofobjects 56 to the computer 4.

Layering

Referring to FIGS. 1 and 12-17, an embodiment system 2 to interface auser with a computer 4, which may implement aspects of the previouslydescribed system 2 or any other embodiment disclosed herein, includes auser interface device 10 including one or more input elements 44 tointerface the user with the graphical environment 42.

A memory (not shown) can be communicatively coupled (e.g. forinformation transfer) to said processor 26 and/or 34 to store aplurality of different layers 64 to represent the user interface device10 in the graphical environment 42 and/or a field proximal the userinterface device 10 and/or a body part 6 interfacing with the interfacedevice 10. The processor 26 and/or 34 may provide in said instructionsone or more of the layers, e.g. the one or more different layersincluded in the instructions are selected from the plurality ofdifferent layers 46.

As used herein the term “layer” in reference to the graphicalenvironment may refer to a two or three-dimensional representation of anobject in the real-world and/or the virtual reality environment. Thelayer may be rendered from an image of a real-world object (which may beprovided from the camera system) or otherwise representative, includingby augmentation.

Referring to FIG. 12, a real-world representation of the user interfacedevice 10, which implements aspects of the previously described userinterface device 10 or any other embodiment disclosed herein, isarranged as a computer keyboard. The keyboard 10 includes input elements44 of the I/O system 38 to interface the user with said graphicalenvironment 42. The input elements 44 are arranged as mechanicallyactuatable keys. In variant embodiments there may be any number of inputelements, e.g. one or more. Moreover, said input elements may take otherforms include those disclosed herein. Input elements can be included inperipherals which provide enhanced functionality, such as a wheel thathas input elements, or a joystick that has buttons and other inputelements.

In certain embodiments, a user input device 10 is graphicallyrepresented as a series of “overlays” that are displayed in the virtualreality environment to the user on top of one another. The layersmodularize the processing and modification of the virtualrepresentation. A layer could be the surface area surrounding akeyboard, a layer can be the enlarged alphanumeric characters, a layercould be highlighting the alphanumeric characters the user is hoveringover, a layer could be the hands of the user (semi-transparent, maybeonly parts of the hands, etc.). The layers can be contextually relevant,meaning they can change based upon current virtual reality context orother stimuli. In certain embodiments, a user, in the VR environment,may separate the peripheral from the layers, expand the layers to seewhat has been augmented, and/or customize the layers based upon theirpreferences.

Layers which fit to the shape of the real physical keyboard can be addedto augment the keyboard rendered in the virtual environment and makekeycaps more visible and useable. Layers allow for a level of visualfeedback (lighting up the augmented image of the key, in a variety ofways) when a key is pressed, as well as transforming the keyboard into acompletely new augmented offering—like the emoji keyboard mentionedabove, as well as solving localization issues.

The layers can be superimposed over a video image of the actual keyboardor other device from a camera. The alignment can be enhanced bysearching for unique shapes on the keyboard, such as corners, a specialalignment feature such as a cross-hair on a corner, or the activation ofelements of the keyboard, such as lighting a key, frame, or other partof the keyboard and informing a tracking system of what has beenilluminated. A keyboard or other device can be manufactured withfeatures that enhance detection and alignment, such as lighting aroundthe edges, high contrast between the keys and frame, etc. Also,particular key presses can be relayed back to the tracking system, sothe fingertip and key being depressed can be identified from the imageand used to update the alignment. Precision alignment can be prioritizedin some embodiments for X, Z dimensions, and yaw, pitch rotation. Thereis typically less impact on alignment for the Y dimension, and roll isconstrained for flat surface usage.

In certain embodiments, an opacity or transparency of each layer can beadjusted initially or dynamically to provide an optimum image. Forexample, key indicia may be made bright enough to show through a fingerwhen it is pressed, but not otherwise.

Referring to FIGS. 13-15, the layers 64 to represent the keyboard 11 caninclude one or more of the following: an indicia and input element layer68 to represent one or more of the input elements and the indicia on theinput elements as shown in FIG. 13 (alternately a separate indicia layer66 may be used to show the indicia separately from the input element,such as a key indicia, as shown in FIG. 14); a feedback layer 70 toprovide user feedback of a state of the one or more input element; adevice body layer 72 to represent a body of the keyboard 10. A proximallayer 74 represents a field proximal to the user interface device 10(e.g. the work surface, desktop, a mat or pad). It will be understoodthat the embodiment layer representation can be implemented with otheruser interface devices than a keyboard, including those disclosedherein.

The indicia layer 66 includes indicia to represent an input functionassociated with the or each input element 44 (the input function may bestored by the memory coupled to the processor). Referring to FIG. 14 theindicia layer 66 comprise indicia as alphanumeric and other symbols.Referring to FIG. 15 the indicia layer 66 comprise indicia as emoji. Inembodiments, other indicia of other indicia layers 66 include gamingmacros (G-keys), ‘hot’ keys.

Referring to FIG. 14, some of the indicia of the indicia layer 66 cancomprise the equivalent indicia to the indicia of the real-world userinterface device 10 of FIG. 12 (e.g. the alphanumeric symbols) and someof the indicia of the indicia layer 66 are augmented or changed based oncontext, such as automatic triggers or user commands. The augmented orchanged indicia thus comprise different indicia to the indicia of thereal-world user interface device 10 of FIG. 12 (e.g. the windows key andthe enter key). Referring to FIG. 15 the emoji are different indiciathan the indicia of the real-world user interface device 10 of FIG. 12.Accordingly, in embodiments, the equivalent and/or different indicia canbe mapped onto the indicia layer as on the real-world user interfacedevice 10.

In embodiments where an input element does not provide any function inthe virtual reality environment no indicia may be provided in theindicial layer 66, e.g. referring to FIG. 15 the input elementsperipheral to those associated to the emoji do not have indicia. Forexample, the edges of the case of a keyboard may not have keys. In oneembodiment, non-functional areas may be changed in appearance or havefunctionality added. Also, keys not currently usable in a given mode maybe modified. For example, if the windows key isn't relevant, the windowskey can be changed to something the user might deem useful, which inturn inherently means the functionality of the keyboard needs to bemodified as layers are incorporated.

The real-world user interface device 10 may not include any indiciaassociated with the or each input element, wherein, as part of thevirtual reality environment, the indicia are specified by the indiciallayer 66. Examples of such a user interface device 10 include a keyboardthat has keys without any symbols or a touch pad that has the indiciallayer 66 specified to represent keys.

Other user input devices or peripherals may be represented by the sameor different layers. For example, a mouse may be represented by an inputelement layer to represent the mouse buttons and scrolling wheel. Afeedback layer modifies the appearance of the input element layer, suchas by highlighting the button pressed, enlarging the button or scrollingwheel, a device body layer represents a body of the mouse, which canhave its color, shape, etc. changed in the virtual reality environment.These layers can be curved, to correspond to the shape of the mouse. Thelayers can be continuous or discontinuous, and each layer can be brokeninto different, separately controlled and displayed portions, ormultiple layers could be combined. A proximal layer represents amousepad or work surface. Similar layers can be provided for a joystick,gamepad, steering wheel, or any other user input device, peripheral orobject.

It will be understood that the function of the input element can belinked to the associated indicia of the indicial layer, e.g. asdifferent indicial layers 66 are transitioned the function iscorrespondingly transitioned. For example, if alphanumeric keys arechanged to emojis, the pressing of those keys by the user causes anemoji to be typed, not the alphanumeric character. Other augmentationsor changes, such as highlighting or enlarging a key, do not change thefunction of the key.

In one embodiment, the layers are deconstructed from a camera image,suitably manipulated and augmented, then combined again for display inthe virtual reality environment. This may be done as an initializationstep, or periodically, such as after detecting the location of the userinput device has changed. Alternately, stored images of one or morelayers may be combined with the camera image of the input device, basedon the obtained model number of the user input device. In oneembodiment, the camera image is only used to determine location andmovement, and the layers are entirely generated from a pre-stored image.

It will be understood that the input element layer 68 may be configuredin a similar manner to the indicia layer 66, e.g. with some or all ofthe input elements 44 of the real-world user interface device 10represented in the input element layer 68. In embodiments, the inputelements may be represented in a different manner to their arrangementin the real-world, e.g., some input elements may not be represented inthe input element layer 68 or some input elements with the same functionmay be combined as a larger input element in the input element layer 68.

The feedback layer 70 may provide user feedback of a state associatedwith of the one or more input element 44 that may be represented in theinput element layer 68 and/or the indicia layer 66. The state mayinclude that the associated input element 44 has been interfaced with(e.g. selected) or is contemplated for interfacing with by the body partof user 6. Interfacing and/or contemplated Interfacing of the inputelement 44 may be determined by the associated sensor of the I/O system38 and/or the sensor system 12 (including a sensor system operativelyconnected to the user interface device 10, which may include a camerasystem and/or capacitive sensing system).

As user herein the term “contemplated interfacing” or “contemplated forinterfacing” or “contemplated for selection” may refer to a user bodypart arranged in operative proximity to select or otherwise interfacewith an input element, e.g. a finger of a user hovers over a key priorto selection.

Referring FIG. 14, the feedback layer 70 can include a highlighted andenlarged representation of an associated input element and indicia(represented by the input element layer 68 and the indicia layer 66) ofan input element that is contemplated for section by a body part(represented by a body part of a body part layer 76). In embodiments,the feedback layer 70 may implement various visual indicators ofinterfacing or contemplated interfacing, examples of which include oneor more of the following: highlighting; lowlighting; enlargement;reduction; a color change, of all or part of a graphical representationof an associated input element and/or indicia.

The device body layer 72 may represent any portion of a body of the userinterface device 10, including a peripheral surface (e.g. which forms aplanform visible in use of the device to the user) and/or across-sectional portion of the device. A planform is the contour of anobject as viewed from above.

Referring FIGS. 14-16, the device body layer 72 can represent a planformsurface, which is visible in use of the body of the keyboard 10, and isarranged around the input elements 44 thereof.

In embodiments, when representing other user interface devices 10, thebody may include a wheel of an electronic steering wheel or a stickand/or base of a joy stick, a mouse, speaker, or any other user inputdevice, peripheral or object. In certain embodiments, the device bodylayer 72 can represent only an operative portion of the user interfacedevice 10, e.g. only the numerical entry portion of the keyboard 10 isrepresented. In certain embodiments, the body layer 72 can represent analternative device other than the real-world user interface device 10,e.g. the keyboard 11 can be represented as an alternative peripheraldevice such as a game controller.

Referring to FIGS. 13-16, the layers 64 can include a proximal layer 74to represent a field proximal the user interface device 10 as definedherein (e.g. the work surface, a mat or pad, etc.). The proximal layer74 may thus include a representation of any of the previously describedobjects that may be arranged in said proximal field. In embodiments, theproximal layer 74 may include environmental media 77. Referring to FIG.16, the environmental medial 77 includes light and/or shadows present inthe real-world environment, including from daylight, room lighting orother ambient light. The objects and environmental media 77 and may bedetermined by the sensor system 12.

Referring to FIGS. 14-16 the layers 64 include a body part layer 76 torepresent a body part 6. The body part may be represented in the bodypart layer 76 as discussed previously. A layer can be provided that isresponsive to motion, such as an interactive space which allows users tomanipulate objects (e.g., totems or VR images), using the sensor system.

Referring to FIGS. 14-16, various layers 64 can be superposed on eachother. Referring to FIG. 16 the body part layer 76 can be represented aspartially transparent, through which other layers are visible.Accordingly, in embodiments, one or more of the layers 64 may byrepresented as partially transparent, wherein: a partially transparentlayer may be superposed on another layer (which may be another partiallytransparent layer or a solid layer); and/or a layer (which may be asolid layer) is superposed on a partially transparent layer.

In certain embodiments, one or more of the layers can representdifferent positions of a depth direction associated with the userinterface device 10, e.g. a solid base layer is superposed by otherlayers of increasing transparency based on depth. Alternately, asdescribed above, the layers correspond to different functions, such asinput elements or keys, which may extend down inside another layer, suchas a device body layer.

In certain embodiments, various combinations of one or more of thelayers 66-76 or other layers can be provided in non-transitorycomputer-readable media accessed by instruction running on a processor.In certain embodiments, one or more of the layers 64 are stored on amemory of the user interface device 10, and may be transferred to thecomputer 4 for processing. In certain embodiments, one or more of thelayers 64 can be stored on a memory associated with the computer 4 andcan be obtained following identification of the user interface device10, e.g. by means of a database, which may have a key-value or othersuitable implementation. The user interface device 10 may be identifiedby techniques including one or more of the following: transmitting aunique identifier to the computer 4 (which may be used as the key); byimage processing of an image, which may be obtained by the camerasystem; a positioning system as previously discussed.

In certain embodiments, the processors 26, 34 may partially or fullygenerate one or more of the layers based on images of the user interfacedevice 10, which may be obtained from the camera system. In certainembodiments, the user interface device 10 may be adapted to facilitatesaid generation of the one or more layers. Said adaption may include theformation of a non-reflective surfaces and/or surfaces which areconvenient to identify.

In certain embodiments, the one or more layers 64 can be arrangedrepresentative of the real-word arrangement of the associate object. Thebody part layer may include representative arrangement of the body part6 based on the determined real-word arrangement of the body part 6. Thebody part of the body part layer and the layers representing the userinterface 10 may be arranged an equivalent position to that in thereal-world, including in the field of view in the graphical environment42 equivalent to that in the field of view in the real-worldenvironment. The sensor system 12 for determining the real-worldarrangement of the body part 6 and/or user interface device 10 may beoperatively connected to the user interface device 10 as previouslydiscussed.

In certain embodiments, one or more of the layers 64 can be usercustomizable, an example includes the indicia of the indicial layer 68may be selected by the user and/or the configuration of the feedbacklayer 70 may be selected by the user.

In certain embodiments, one or more of the layers 64 are fitted to thereal-world arrangement of a portion of the user interface device 10. Anexample includes the Input element layer 68 being fitted to keys of akeyboard and the body layer 72 fitted to the periphery of a keyboard.

In embodiments, the instructions may include or transition one or moreof the layers 64 based on a trigger. “Transition” may refer to changingthe representation of one or more layer from a first representation to asecond representation (e.g. in the instance of the indicia layer 66 afirst representation may include the indicia as alphanumeric and othersymbols as shown in FIG. 14 and the second state may include the indiciaas emoji as shown in FIG. 15). In certain embodiments, the trigger caninclude one or more of the following (which may be determined by theprocessor 26, 34).

-   -   A configuration of the user interface device 10, which may        include determination of a type of user interface device 10. The        type of the user interface device 10 may be determined by        identification of the user interface device 10 as discussed        previously. The type of the user interface device 10 may relate        to the form factor of said device.    -   An arrangement of the user interface device 10, which may        include a location of the user interface device 10, examples of        locations include the arrangement of the user interface device        10 on a work surface (e.g. a keyboard) or held by the user (e.g.        a game controller), Arrangement may include the real-world        arrangement of the user interface device 10, including an        orientation of the device. The orientation may be relative a        reference point (e.g. the HDM 8 or other element of the user),        examples include an orientation of the device which may be        defined by an alignment of a longitudinal axis of the user        interface device 10 (e.g. aligned vertically or pointing away        from the user),    -   A proximity of an object associated with the system 2. Proximity        may be with reference to the user interfaced device 10, e.g. the        proximity of the object to the said device 10 (which may be        determined using the sensor system 12 arranged on the user        interface device 10). The object may be as previously defined.        Proximity may include the previously defined proximal field. An        example includes the user interface device 10 arranged in        operative proximity an object arranged as an interoperable user        interface device, wherein input element functionally its        transferred/shared between the devices.    -   A contextual event, which may be related to the virtual reality        environment (e.g. a particular stage in a VR game) or other        computer implemented notification including the receipt of a        message or email). A contextual event may also include the        environmental media 77,    -   A user interaction with the system, which may include a gesture        that may be made by fingers/hand. The gesture may include a        characteristic movement of the hand, including a hand swipe or        finger moment (e.g. to press an input element). The gesture may        include a characteristic shape made by the hand, such as natural        hand movements for grasping, pointing, pulling apart, etc. An        example of a gesture includes a moving their finger to make a        selection of an input element 44 of the user interface device        10, in other embodiments, the user interaction includes input        determined from the I/O system 38 of the user interface device        10 via the user interfacing with of one or more of the input        elements 44.    -   User configured, which may include the user configuring the        configuration of one or more of the layers 64 and/or their        transition.

Inclusion and transition of the one or more of the layers 64 in theinstructions based on a trigger will now be exemplified for the varioustriggers defined above.

In certain embodiments, modification or generation of one or more of thelayers 64 may be triggered based on the user interface device 10 beingidentified, e.g. as a game controller or keyboard or other userinterface device. One or more of the indicia layer 66, input elementlayer 68, feedback layer 70, device body layer 72 may be configured tooverlay the real-world device.

In certain embodiments, instructions for one or more of the layers 64may be triggered based on the user interface device 10 determined asarranged on a work surface, e.g. the proximal layer 74 is arranged torepresent the work surface, as previously discussed.

In certain embodiments, for a keyboard 10 (or other user interfacedevice) determined as transitioned from abutment with a work surface tobeing held by a user, the instructors for the one or more of the indicialayer 66, may be accordingly transitioned (e.g. to represent inputelements 44 with reduced functionality) since a user may be unable touse both hands to type.

In certain embodiments, instructions for one or more of the layers 64may be triggered based on the other user objects 56 determined as in theproximal field the user interface device 10 (e.g. by the previouslydescribed sensor system 12 arranged on the user interface device 10). Incertain embodiments, wherein the objects 56 are interoperable userinterface devices, one or more of the indicia layer 66, input elementlayer 68, feedback layer 70 of the user device 10 may be transitioned asinput function is transferred to/from the interoperable user interfacedevices, e.g. a joystick wheel is arranged proximal the keyboard,wherein the keyboard transfers/shares functionality of the cursor keysto the stick).

In certain embodiments, the proximal layer 74 is transitioned torepresent objects 56 therein. In an embodiment one or more of theobjects 56 are represented by layers 64 as defined herein in a similarmanner as for the user interface device 10.

In certain embodiments, the virtual reality environment or notificationthat requires alphanumeric entry and one or more of the indicia layer66, input element layer 68, feedback layer 70 (or other layer) istransitioned to represent input elements for said alphanumeric entry.

In certain embodiments, a gesture (e.g. a swipe) is used to transitionthe indicia layer 66 as alphanumeric and other symbols as shown in FIG.14 to the indicia as emoji as shown in FIG. 15. It will be understoodthat various gestures may be used to transition of one or more of thelayers 64 to various representations.

Referring to FIG. 17, process of interfacing a user with a computer,which may be implemented by the processor may include, at block 78,selecting one or more layers from a plurality of different layers 64 torepresent a physical user interface device 10 in graphical environment42 of a virtual or augmented reality virtual reality environment. Theprocess may include, at block 80, generating instructions to displaysaid one or more selected layers in the graphical environment 42. Atblock 82, the method may include determining a trigger and in responsetransitioning one or more of the layers included in the instructions.

Peripheral-Centric Augmented Workstation Environment

In some embodiments, a peripheral-centric augmented reality workstationenvironment can be realized using an AR/VR system. Some systems,including some of those described above, may be configured to render anaugmented reality workstation environment that is aligned relative to auser. For instance, some embodiments can render video content in anaugmented reality or virtual reality environment based on a location ofa user or an HMD (see, e.g., FIG. 1D). That is, virtual displays,virtual objects, and/or other AR/VR rendered virtual content can berendered to appear to be located at certain positions when displayed toa user.

A peripheral-centric augmented reality workstation environment may bealternatively configured to distribute various content relative to aperipheral device (e.g., keyboard, mouse, stylus, hub, etc.), ratherthan relative to an HMD. By way of example, some embodiments of anaugmented reality workstation environment may be configured to determinea location of a physical peripheral input device (e.g., keyboard) withina physical environment and determine, based on the location of theperipheral input device within the physical environment, a locationand/or orientation of a virtual display (e.g., via an MID) to render toa user of the peripheral input device. A virtual display can beseparated and distinct from the corresponding peripheral input device.For example, a virtual “television” or other display can be rendered asappearing to be located on a wall or at a distance for a user of anperipheral input device. In some aspects, the orientation and/orlocation of the display can remain at a fixed, perceived spatialrelationship with respect to the peripheral device as the peripheraldevice moves within the physical environment. For instance, if a userpicks up the peripheral device from one location and moves it to asecond location, the display may maintain its perceived fixed positionrelative to the peripheral device. Some displays may be “sticky,” suchthat a small movement of a peripheral device (e.g., resituating akeyboard by 20 cm on a surface) may not cause a display to move, butlarger movements (e.g., 3 m) may prompt the display to move in a fixedmanner, as further discussed below.

Some embodiments may employ additional interactive displays that mayalso be rendered in locations relative to the peripheral device. Thesedifferent types of displays can be rendered in different designatedareas, or “zones” depending on their intended content and use. Forinstance, content that is intended to be interacted with by a user maybe configured in a location that is easily accessible by the user (e.g.,on or near the peripheral device), while content that is typically onlyintended for viewing may be configured farther from the peripheraldevice at a preferable viewing distance. The following embodimentsdepict these aspects in further detail, and one of ordinary skill in theart would appreciate that any of the embodiments depicted and/ordescribed throughout this document, or portions thereof, can be combinedin any suitable manner, and accordingly any one embodiment should not beinterpreted as a limiting combination of features.

FIG. 18 shows a simplified diagram of a peripheral-centric augmentedreality workstation environment (AWE) 1800, according to certainembodiments. AWE 1800 can include computing system 1810, display system1820, sensor system 1840, and peripheral device(s) 1830(1-n). Computingsystem 1810 can be capable of providing an augmented/mixed/virtualreality environment to a user via display system 1820. Display system1820 can be embodied as an HMD, virtual reality display, holographicimaging device, or other display capable of providing computer-renderedimages to a user. The one or more peripheral devices 1830 (1-n) can be aphysical user interface device configured to enable a user to interfacewith the augmented workstation environment facilitated by computer 1810.As indicated above, typically one of the peripheral devices 1830 can beused as a reference point to arrange one or more virtual displays to auser. Sensor system 1840 may sense and provide position and orientationinformation to computer 1810, including a location of the one or moreperipheral devices, the user, body parts of the user (e.g., location ofhead, hands, arms, etc.), physical characteristics of the physicalenvironment around the one or more peripheral devices (e.g., location ofsurfaces, walls, objects, obstructions, etc.), and the like, as furtherdescribed below.

Computer 1810 can include a host processor, which may include amicroprocessor, multiple processors and/or co-processor chips, and/ordigital signal processor (DSP) capability, or the like. A system clock(not shown) may be coupled to or part of host processor 30 to providetiming data. Computer 1810 may include an audio system including audiooutput devices to provide audio to a user, as would be appreciated byone of ordinary skill in the art with the benefit of this disclosure.Display system 1820 may display images of a simulation, gameenvironment, operating system application or other images associatedwith the simulation. As shown in the forthcoming figures andcorresponding description, display system 1820 can render images of anaugmented workstation environment. Display system 1820 may be part of anHMD, forming/displaying the augmented workstation environment, ordisplay system 1820 may be a separate secondary display device (e.g.,holographic display device) to allow configuration of the system by theuser without needing to wear the HMD, or configuration of the system byany other person, or to allow a secondary non-VR display of the virtualreality environment, e.g., for observational, run-time configurational,safety or any other purpose(s). Computer 1810 may include other knowncomponents, such as random access memory (RAM), read-only memory (ROM),and input/output (I/O) systems, and the like, as would be appreciated byone of ordinary skill in the art.

Computer 1810 may implement an application program, which may be asimulation program for generation of the augmented workstationenvironment. The user may interact with the program via peripherals 1830(e.g., keyboard, mouse, stylus, etc.). The application program mayinclude multiple rendered displays arranged relative to a peripheraldevice to display an office productivity environment, a gamingenvironment, an interactive digital location (e.g., home, virtual store,sporting arena, etc.), medical procedure simulation, computer-aideddesign applications, or other type of virtual arrangement of interfaces.The application program may comprise or access an external database,such as over a network. The application program may be implemented asone or more modules or other functional units. Herein, for simplicity,operating systems such as Windows™, Android; IOS; MS-DOS, MacOS, Linux,etc., are also referred to as application programs as may be devicedrivers for hardware associated with the computer. Typically, theapplication program can be capable of providing instructions for thegeneration of a graphical environment on display system 1820. It mayprovide images to be displayed on display 6 of HMD 18 and may outputother feedback, such as auditory or vibration (haptic) signals. Theapplication program may be operable to check for input signals fromperipherals 20 and provide the corresponding output. The applicationprogram may interface with the HMD 18 and/or other peripherals 20 via adevice driver, whereby the device driver communicates with the devicethrough electronic circuitry of I/O system 38.

Computer 1810 may be embodied in a personal computer, workstation,laptop or server, such as a PC compatible computer, Apple® personalcomputer, smart device (e.g., smart phone, smart watch, etc.), astandalone HMD system, a tablet computer, or other suitable computingsystem. In some cases, computer 1810, as well as the other computersdescribed throughout this disclosure (e.g., computer 2410), mayincorporate aspects of cloud computing for offloading processingfunctions. One of ordinary skill in the art with the benefit of thisdisclosure would understand the many variations, modifications, andalternative embodiments thereof. Computer 1810 may be operable under theWindows™, MacOS™, Unix™, or MS-DOS™ operating system or the like.

In some embodiments, sensor system 1840 may include an object trackingsystem having a camera system, such as one of more of a: 2D camera; a 3Dcamera; an IR camera; a time of flight (ToF) camera, or the like, whichmay utilize CMOS, CCD, IR, or other suitable type of image sensors.Sensor system may further incorporate touch sensing capabilities, whichmay include capacitive-based and/or resistor-based touch sensors (FSR),or the like. Touch sensors generally comprise sensing elements suitableto detect a signal such as direct contact, electromagnetic orelectrostatic fields, or a beam of electromagnetic radiation. Touchsensors can typically detect changes in a received signal, the presenceof a signal, or the absence of a signal. A touch sensor may include asource for emitting the detected signal, or the signal may be generatedby a secondary source. Touch sensors may be configured to detect thepresence of an object at a distance from a reference zone or point(e.g., <5 mm), contact with a reference zone or point, or a combinationthereof. Further aspects of sensor system 1840 are further describedbelow. Some examples of the types of touch/proximity sensors mayinclude, but are not limited to, resistive sensors (e.g., standardair-gap 4-wire based, based on carbon loaded plastics which havedifferent electrical characteristics depending on the pressure (FSR),interpolated FSR, etc.), capacitive sensors (e.g., surface capacitance,self-capacitance, mutual capacitance, etc.), optical sensors (e.g.,infrared light barriers matrix, laser based diode coupled withphoto-detectors that could measure the time of flight of the light path,etc.), acoustic sensors (e.g., piezo-buzzer coupled with microphones todetect the modification of a wave propagation pattern related to touchpoints, etc.), or the like.

Peripheral(s) 1830(1-n) can include a keyboard, computer mouse, audiodevices (e.g., speakers), stylus/touch devices, presenter devices, touchpads, camera-based devices (e.g., a webcam), printers, or the like. Theembodiments that follow describe the use of certain physical peripheraldevices, however it should be understood that these examples are notintended to be limiting, and that one of ordinary skill in the art wouldappreciate how any suitable type or number of peripherals could beintegrated into an augmented workstation environment, as shown anddescribed.

Although certain systems may not expressly discussed, they should beconsidered as part of AWS 1800, as would be understood by one ofordinary skill in the art. For example, AWS 1800 may include a bussystem to transfer power and/or data to and from the different systemstherein. In some embodiments, AWS 1800 may include a storage subsystem(not shown). A storage subsystem can store one or more software programsto be executed by processors. It should be understood that “software”can refer to sequences of instructions that, when executed by processingunit(s) (e.g., processors, processing devices, etc.), cause AWS 1800 toperform certain operations of software programs. The instructions can bestored as firmware residing in read only memory (ROM) and/orapplications stored in media storage that can be read into memory forprocessing by processing devices. Software can be implemented as asingle program or a collection of separate programs and can be stored innon-volatile storage and copied in whole or in-part to volatile workingmemory during program execution. From a storage subsystem, processingdevices can retrieve program instructions to execute in order to executevarious operations (e.g., software-controlled spring auto-adjustment,etc.) as described herein.

Display “Zones” Configured Relative to a Peripheral Device in an AWE

In some embodiments, one or more displays may be rendered in anaugmented reality workstation environment using a physical peripheraldevice as a point of reference. For example, one or more rendereddisplays may be configured within different areas (referred to herein as“zones”) positioned at locations relative to a particular peripheraldevice. As used herein, the zones may be physical areas that contentrendered to a user of an augmented reality workstation may appear to bewithin. Some displays may be intended for visual data consumption andmay be configured farther from the user, while other displays may beintended to provide interactive capabilities and are configured closerto the user (e.g., on or adjacent to the physical peripheral device).The interactive zones may be positioned to enable a user to touch theinteractive displays with a peripheral, appendage, etc. Some examples ofthe use of “zones” in an AWE are further described below.

FIG. 19 shows an example of an augmented workstation environment 1900,according to certain embodiments. The AWE can include a number ofdisplays of varying size and location depending on their intended use.Referring to FIG. 19, certain types of displays are rendered in areasdefined as “zones” (e.g., “zone 1,” “zone 2,” “zone 3”, . . . ), whichare configured at locations relative to a physical peripheral device(e.g., keyboard 1830(1)). The physical peripheral device that is used asa reference point will be referred to as the “reference peripheral.”Each zone may be arranged in any suitable location and may be configuredto span across any suitable area or volume within 3D space relative tothe reference peripheral. In some cases, as the reference peripheralmoves in real 3D space, the zones and corresponding visual content maymove accordingly, relative to the reference peripheral. That is, eachzone may be fixed to a set spatial relationship relative to thereference peripheral, such that the reference peripheral and the one ormore displays rendered in zones 1-3 appear to move as a singularinterconnected system. AWE 1900 may include some or all elements of AWE1800, and certain examples provided herein may refer to both systemsinterchangeably.

In some embodiments, placing the reference peripheral on a surface maycause AWE 1900 to switch to a “wake” state and render the plurality ofdisplays in zones 1-3 in HMD 1820. Placement detection can be achievedin any suitable manner including via a physical switch or sensor (e.g.,pressure sensor, IMU) on the reference peripheral. Alternatively oradditionally, one or more sensors (e.g., visual sensors, ultrasonicsensors, IR sensors, etc.) from sensor array 1840 may be used todetermine when the reference peripheral is placed on a surface.Conversely, lifting the reference peripheral off of a surface may causethe AWE 1900 to switch to a “sleep” or shutdown state. In some cases,lifting and moving a short distance (e.g., 30 cm) may not initiate ashutdown state, as further described below. In certain embodiments, inaddition to the previously described “sleep” and shutdown states, atravel or other state change can occur. For example, entering a travelstate may collapse or otherwise rearrange virtual displays presented toa user of AWE 1900. When the user enters an area wherein they can resumeinteracting with a virtual display using the peripheral input device,the virtual displays may return to a previous position. If, however, theuser has entered a new environment, virtual display(s) may take adifferent position/orientations. For example, if a user moves from anoffice environment to a train, virtual displays may be minimized and/orsuperfluous virtual displays may be removed. Different states may betriggered depending on various criteria. For example, a peripheraldevice may be reconfigured (for example, a collapsible keyboard may becollapsed or a hand held peripheral may be lifted from a work surface).While in different state(s) different input sensors and/or input schemasmay be used. For example, gestures may be relied upon more heavily whilein a travel mode as opposed when working within an office.

Alternatively or additionally, different states may be triggered basedon a proximity detection of a user (e.g., near the referenceperipheral), detecting a user's hand's on the reference peripheral (orother peripheral device in AWE 1900), or the like. The sensing can beperformed by one or more sensors (e.g., vision-based, touch-based,ultra-sound, IR, audio-based, etc.) on the reference peripheral,secondary peripheral devices, computer 1810, display system 1820, or anycombination thereof, as would be appreciated by one of ordinary skill inthe art with the benefit of this disclosure. In some embodiments, thereference peripheral can include tracking features (e.g., sensors, LEDs,fiducial mark(s)) for locating within a physical environment. Forexample, reference peripheral can be tracked from the HMD, from its ownon-board sensors (e.g., via IMU), or a combination thereof. Thus, statechange conditions may be triggered automatically in response toreference peripheral 1830(1) detecting a user's presence, receiving userinputs, detecting placement on a suitable work surface (e.g., based ondimensions and nearby walls, obstructions, open space, etc.), or thelike.

In some embodiments, “zone 1” may be associated with a region on (e.g.,overlaid) or near/adjacent to the reference peripheral (e.g., keyboard1830(1)), as shown in FIG. 19. Zone 1 can be associated with physicalperipheral device(s) that a user may physically interact with. Physicalperipheral device(s) within zone 1 may be augmented with virtualcontent. For example, keyboard 1830(1) may include a number ofdepressible keys but no printed insignia on each key. In someembodiments, a display system (e.g., HMD) may overlay virtualalphanumeric characters over each key in any suitable language. In somecases, the overlay may be contextual such that the virtual alphanumericlayout may be a first set of rendered characters and numbers whenaccessing a first application (e.g., office productivity software), or asecond set of rendered characters and numbers when accessing a secondapplication (e.g., CAD software). Alternatively or additionally,different key layouts may be shown on the reference peripheral inresponse to a physical command (e.g., pressing a function key), or othertrigger (e.g., voice command, a command generated on another peripheraldevice (e.g., mouse 1830(2), stylus 1830(3), etc.). In some aspects,other interactive symbols or non-interactive displays can besuperimposed (overlaid) on reference peripheral 1830(1), includingemojis, application shortcuts, or the like.

Certain embodiments may employ a physical touch sensitive surface (e.g.,touch pad) to receive a user input. Some embodiments may includeadditional sensors in reference peripheral 1830(1) to detect thepresence and/or location of other peripherals 1830(2-n) relative toreference peripheral 1830(1), or define a region within or adjacent tozone 1 that can be designated for receiving a user input (e.g., movementof a finger or stylus along a surface adjacent to reference peripheral1830(1). For instance, vision-based sensors can be used to detect when auser's hand moves within a particular location on a surface adjacent tokeyboard 1830(1) (e.g., a mini-zone adjacent to zone 1). Thus, a sectionof an otherwise inert surface (e.g., a table top) may operate as a highprecision touch sensitive and/or proximity sensing surface. Further, thedesignated touch area may maintain its spatial relationship (e.g.,remain fixed) with respect to reference peripheral 1830(1).

As mentioned above, one or more sensors coupled to reference peripheral1830(1) can be used to track a location of one or more additionalperipheral devices 1830(2-n), with reference peripheral 1830(1)operating as a point of reference. Thus, reference peripheral 1830(1)may operate as a central hub with its own dedicated tracking system.This may present certain advantages over systems that track multipleperipherals from a conventional HMD or lighthouse-based system. Forexample, the additional sensors may provide higher fidelity trackingthan may be possible from HMD-tracking resources. In some cases,offloading the tracking of additional peripheral devices to the hub, andrelaying that tracking information to the computer system 1810 and/ordisplay system 1820 (e.g., to fuse both tracking coordinate systems) mayrelieve the computer system and/or display system 1820 or processingbandwidth. The reference peripheral input device may include featuresfor determining the location of the physical peripheral input devicewithin the physical environment by a tracking system and the trackingsystem can be used to determine an orientation of a physical displayused to render the first virtual display. In some embodiments, thefeatures can be selected from a list including a sensor, an emitter, anda marking. The sensor can be configured to detect or the emitter can beconfigured to emit: visible light, infrared light, ultrasound, magneticfields, or radio waves. In some cases, the physical peripheral inputdevice can include a plurality of the features to enable the physicalperipheral input device to be tracked within the physical environment byany one of a plurality of tracking techniques. The physical peripheralinput device may include an inertial measurement unit (IMU) and thelocation of the physical peripheral input device within the physicalenvironment can be determined using the IMU. In certain embodiments, theorientation of the first virtual display can be determined based on adetermined identity of the user of the physical peripheral input deviceand wherein the orientation of the first virtual display would berendered differently for a differently identified user.

In some embodiments, “zone 2” may correspond to one or more areasadjacent to or near zone 1 that may be designated to render interactivedisplay information within reach of the user. As shown in FIG. 19, zone2 may include interactive tools and content such as tool alterations,mode changes, emoji, chat boxes, application launch buttons, etc. Insome cases, the reference peripheral may include physical areas (e.g.,protruding surfaces) that may be dedicated for the projection ofinteractive displays. The physical areas with virtual interactiveoverlays may include dedicated physical switches, touch sensitivesurfaces and areas with interactive haptics (e.g., over-air haptics,piezo-based haptics, etc.), sensors to track in-air movement above thereference peripheral, or the like, to further augment the userexperience. Zone 2 may be configured on physical areas or in 3D space(as shown in FIG. 19) that is preferably within reach of a user that isaccessing reference peripheral 1830(1). Zone 2 may be configured on thesame plane as reference peripheral 1830(1) or a different plane (asshown). Zone 2 may be a volumetric space, as opposed to a planar region,and can be configured in any suitable manner. In some cases, theboundaries defining the one or more zones can be customized by a user,auto-arranged by AWE 1900 (e.g., based on the visual content), set at adefault size and location, or a combination thereof. Typically, zone 2(and zone 1 in some instances) include a collection of interactableelements that can be interfaced via a user's hands, gaze, voice, acursor, or the like, and may include drag-and-drop functionality (e.g.,locally and with other zones), haptics, touch detection capabilities(e.g, via a dedicated physical dock or designated sensed area), or thelike.

In some embodiments, “zone 3” may correspond to one or more regionsdesignated for visual consumption that may not require user interaction,similar to a physical monitor in some workstation environments.Referring to FIG. 19, zone 3 includes a number of displays distributedalong its defined region. Zone 3 may extend laterally in an arc for awrap-around visual environment, which can be ergonomically beneficialfor the user. In some cases, zone 3 may be a volumetric space to allowcontent to be moved closer or further back from the user. For example,AWS 1900 may detect when a user moves their head closer to the virtualdisplays and cause displayed content in zone 3 to move closer to theuser, which can effectively reduce the distance that the user has tomove forward. Alternative or additionally, detecting that the user(e.g., HMD) is moving closer to zone 3 may cause content in zone 3 toauto zoom at a proportional amount relative to the HMD movement. In somecases, the user may be simply moving their seat closer or farther formthe reference peripheral (thus, moving the HMD a similar distance),which may cause AWE 1900 to move content in zone 3 (or from other zones)closer or farther to maintain a viewing distance. Note that in suchcases, the zones are still projected relative to the referenceperipheral and not the HMD, however the HMD may be a factor isdetermining where to position the one or more zones for an improvedviewing experience. Zone 3 can typically include multiple displaysdistributed thereon, interactive 2D and 3D displays/objects, interactionvia gaze, cursor, voice, and the like.

In some embodiments, a “zone 4” may be defined as an area (e.g., planeor volumetric area) to share virtual content to make for a collaborativeenvironment between users. For example, each user may have their ownreference peripheral (e.g., keyboard) and corresponding zones 1-3,however a zone 4 may be shared by all. Users may select and move contentto and from zone 4 using a mouse, stylus, or other interactiveperipheral device.

The location of zones may be determined by computer 1810, by referenceperipheral 1830(1), by display system 1820, or a combination thereofusing sensor resources from any resource of AWE 1800. For example,reference peripheral 1830(1) may utilize on-board sensor or leveragesensor resources from computer 1810 and/or display system 1820 to pollits environment and determine where boundaries, open areas, surfaces,and obstructions are located, for instance. In some cases, contentpresented to the user via the rendered interactive virtual display canbe selectable by the user to modify an operation corresponding to use ofthe physical peripheral input device by the user, and where theinteractive virtual display can be rendered to appear to the user to bein proximity to the physical peripheral device. The operationcorresponding to use of the physical peripheral input device by the usermay include changing a command corresponding to actuation of an inputelement of the physical peripheral input device, changing an appearanceof the physical peripheral input device, an appearance of an inputelement of the physical peripheral input device, and/or changing arelationship between one or more rendered virtual displays and thephysical peripheral input device. In some cases, the orientation of theinteractive virtual display can be rendered to appear to the user to beat a location that is integrated on or within a periphery of thephysical peripheral input device, or the interactive virtual display canbe rendered to appear to the user to be located on an area of thephysical peripheral input device including features that supplementfunctionality provided by the interactive virtual display. In someembodiments, the features may include haptic feedback generated by ahaptic feedback generator integrated within the physical peripheralinput device to provide haptic feedback in response to user interactionwith the interactive virtual display. The features can include aphysical input element and where the content of the interactive virtualdisplay is rendered to appear to the user to be selectable by the userby actuation of the physical input element. The interactive virtualdisplay can be rendered to appear to the user to be located in an areain proximity to the physical peripheral input device. The content can bedetermined to be selected by the user, without physically contacting thephysical peripheral input device, by a sensor of the physical peripheralinput device. In some cases, content presented to the user via arendered display can be selectable by the user of the second physicalperipheral input device to modify an operation to use of the secondphysical peripheral input device by the user of the second physicalperipheral input device, and the second interactive virtual display canbe rendered to appear to the user of the second physical peripheralinput device to be in proximity to the second physical peripheral inputdevice. In certain embodiments, the interactive virtual display can berendered to appear to the user to be at a spatial location determined,at least in part, based on a biomechanical model of the user to enablethe user to reach the interactive display with an appendage

Environment Detection and User Profiles

In some embodiments, AWE 1900 may auto-adapt to various usage scenariosand/or media content. For example, upon detection that the referenceperipheral is placed on a desk, a 3 zone, 3 monitor mode of operationmay be rendered, as shown for example in FIG. 19. In another example, auser may place the reference peripheral on a small surface in closequarters, such as on a seat tray on an airplane or in a coffee shop. Inthese cases, zones may be configured such that the user is not lookingdirectly at other people in the same line-of-site, visually distributedwithin a smaller radius relative to the reference peripheral, change thenumber of zones, limit the size of content, or the like. For instance,in the coffee shop example, zones 1 and 2 may remain relativelyunchanged, however zone 3 may be directed downward, limited to a smallerarea, and orientated at a different viewing angle relative to thereference peripheral. The sensing of the surrounding physicalenvironment can be performed by one or more sensors (e.g., vision-based,touch-based, ultra-sound, IR, etc.) on the reference peripheral (e.g.,operating as a hub, as further described below), secondary peripheraldevices (non-reference peripherals), computer 1810, display system 1820,or any combination thereof, as would be appreciated by one of ordinaryskill in the art with the benefit of this disclosure.

In some embodiments, a user profile may be configured for each user,which may include content display locations, control schemas, and thelike. For example, a first user may be relatively short and each zonecan be configured in a location that is ergonomically beneficial to thatuser's relative size. Zone 2 may be positioned closer to the referenceperipheral to accommodate the user's corresponding biomechanical range.Likewise, a second user that is relatively tall may have a user profilethat accommodates that user's relative size. In some cases, a userprofile may arrange one or more zones for a particular environment, asdescribed above. A first user may have a first profile for office use(e.g., large surface area with far walls) and a second profile forworking on a bus (e.g., close quarters and downward facing viewingarea). Thus, user preferences and/or the work environment (e.g., walllocation, lighting, proximity of other people, etc.) may be used toestablish different working profiles to better optimize the augmentedworkstation environment. In some embodiments, aspects of AWE 1900 candetect a position/location of the user relative to reference peripheral1830(1) and may identify zones for human ergonomic envelopes (e.g., headmovement) that may have a different response to certain content. Forinstance, content may snap into ideal zones, or a wireframe outlines maybe superimposed on an area/volume to highlight an ergonomicallypreferred configuration different from a current arrangement of zones.In some cases, zone placement may be user defined. For instance, aseries of questions and/or displayed arrangements may be presented to auser to make decisions on how zones will be displayed, what content willbe included in each zone, how zones can affect one another (e.g.,whether or not content may be moved freely from one zone to the next),how to arrange zones in certain work environments, etc.,enabling/disabling interactive controls based on applications (e.g.,word processing may disable object manipulation controls, as one mightuse in a CAD tool). A user may determine how zones are shared betweenusers and what privacy policies to apply. One of ordinary skill in theart with the benefit of this disclosure would appreciate the manyvariations, modifications, and alternative embodiments.

Reference peripheral 1830(1) and/or peripheral devices 1830(2-n) caninclude one or more controls for interacting with interactive content,including but not limited to depth wheels, touch sensors,buttons/switches, in-air use, etc., which may adaptively change theirfunctions depending on the zone that the user is interacting with. Forexample, in response to a user moving their gaze from zone 1 to zone 3(e.g., detected by HMD), a cursor may snap from zone 1 to zone 3 andchange one or more control functions (e.g., functions assigned to abutton, scroll wheel, etc.) based on the type of content configuredthereon (e.g., interactive media, viewing-only media, etc).

In certain embodiments, content may change when moved between zones. Forexample, content displayed in zone 2 that may be configurable andinteractive may become only viewable when moved to zone 3.

“Sticky” Zones and Expanding/Collapsing Zones and in an AWE

FIG. 20 shows how content zones may track with respect to a referenceperipheral, according to certain embodiments. As described above, eachzone in an AWE may be arranged in any suitable location and may beconfigured to span across any suitable area or volume within 3D spacerelative to the reference peripheral. In some cases, as the referenceperipheral moves in real 3D space, the zones and corresponding visualcontent may move accordingly, relative to the reference peripheral. Thatis, each zone may be fixed to a set spatial relationship relative to thereference peripheral, such that the reference peripheral and the one ormore displays rendered in zones 1-3 appear to move as a singularinterconnected system. However, some embodiments may employ a “sticky”spatial relationship the generally tracks the movement of the referenceperipheral, but not necessarily for every type of movement. That is,instead of tracking directly with the peripheral device, there can be athreshold area/distance that the peripheral device can be moved withoutcertain content moving in proportion. For example, consumption displaysconfigured in zone 3 may remain unmoved when a corresponding referenceperipheral is moved below a threshold distance (e.g., 30 cm radius),rather than moved in a fixed arrangement as described above. In anotherexample, a user may want to move a reference peripheral out of the wayto create an open space on a surface immediately in front of the user touse as an interactive surface to touch or write on (e.g., via a stylus).

Referring to FIG. 20, window A shows an AWE with an interactive dock inzone 2 and three virtual displays rendered in zone 3. As the referenceperipheral is moved within a threshold distance (e.g., within avolumetric area), the displays in zone 3 remain unmoved. This may beuseful when a user may resituate their peripheral devices to a betterlocation on a desk, on their lap, where the reference peripheral ismoved to accommodate more peripheral devices in a limited space, or thelike. Any of zones 1-3 (and shared zones, e.g., zone 4) can beconfigured to have a “sticky” spatial relationship with the referenceperipheral, as would be appreciated by one of ordinary skill in the artwith the benefit of this disclosure.

In some embodiments, the state of one or more displays configured on anyof zones 1-3 can be stored and collapsed into a more compactarrangement. This may be useful when a user moves the referenceperipheral from one location to the next, but does not necessarily wantto power down the AWE, or the user may be confined to smaller quartersand cannot expand and number of displays beyond a limited area. Forexample, referring to FIG. 21, window A shows an AWE with an interactivedock in zone 2 and three virtual displays rendered in zone 3. Inresponse to the reference peripheral being lifted, the displays maycollapse into an interactive stack, as shown in window B. Conversely,when the reference peripheral moves back to a suitably sized surface (orother approved location), the displays may expand back to their originalsaved configuration. Any suitable trigger may be used to cause theexpansion and/or collapse of the AWE including lifting the referenceperipheral, selecting a “collapse/expand” function within the AWE, orother method. Further, any arrangement of collapsed screens from anyzone can be configured and customized to preference. Some embodimentsmay cause one or more rendered displays to collapse to a larger orsmaller area than shown in FIG. 21. The extent to which a number ofdisplays may be collapsed may further depend on the detected physicalenvironment (e.g., location of walls, obstructions, otherpeople/devices, etc.) In some cases, the collapsed displays may still beaccessible. For example, when collapsed, displays originally configuredin zone 3 may be moved closer to the user for easier access, so the usercan flip between displays (e.g., via suitable gesture, such as swiping),interact with content, or the like. One of ordinary skill in the artwith the benefit of this disclosure would appreciate the manyvariations, modifications, and alternative embodiments thereof.

FIG. 22 is a simplified flow chart showing aspects of a method 2200 foroperating an augmented workstation environment, according to certainembodiments. Method 2200 can be performed by processing logic that maycomprise hardware (circuitry, dedicated logic, etc.), software operatingon appropriate hardware (such as a general purpose computing system or adedicated machine), firmware (embedded software), or any combinationthereof. In certain embodiments, method 2200 can be performed by aspectsof computer 1810, reference peripheral 1830(1), display system 1820, ora combination thereof.

At step 2210, method 2200 can include determining a location of aphysical peripheral input device (e.g., reference peripheral) within aphysical environment, according to certain embodiments. The location ofthe reference peripheral can be determined using any resource of sensorsystem 1840, which may include one or more sensors of referenceperipheral 1830(1), display system 1820, computer 1810, or anycombination thereof.

At step 2220, method 2200 can include determining, based on the locationof the reference peripheral within the physical environment, anorientation of a display to render to a user of the referenceperipheral, wherein content presented to the user via the rendereddisplay can be modifiable by the user via the reference peripheral. Theorientation of the display may refer to the direction the rendereddisplay faces relative to the reference peripheral, the distance thedisplay is configured relative to the reference peripheral, or acombination thereof.

At step 2230, method 2200 can include determining a type of the rendereddisplay, according to certain embodiments. For example, the display maybe configured for viewing content (e.g., zone 3), interactive content(e.g., zones 1-2), or the like. In some cases, the type of the displaymay affect the orientation/location of the display.

At step 2240, method 2200 can include determining a type of spatialrelationship of the rendered display, according to certain embodiments.For example, the spatial relationship may be fixed (e.g., the distanceof the display relative to the reference peripheral does not change whenthe reference peripheral is moved), or may employ “sticky” displays thatmove relative to the reference peripheral for movement greater than athreshold value (e.g, 30 cm). Any suitable threshold value can be used(e.g., 1 cm-5 m), as would be appreciated by one of ordinary skill inthe art. In some embodiments, the reference peripheral can be akeyboard, hub (as further described below), or other suitable physicalperipheral input device.

It should be appreciated that the specific steps illustrated in FIG. 22provide a particular method 2200 for operating an augmented workstationenvironment, according to certain embodiments. Other sequences of stepsmay also be performed according to alternative embodiments. Furthermore,additional steps may be added or removed depending on the particularapplications. For example, some embodiments may only include methodsteps 2210-2220. Any combination of changes can be used and one ofordinary skill in the art with the benefit of this disclosure wouldunderstand the many variations, modifications, and alternativeembodiments thereof.

FIG. 23 is a simplified flow chart showing aspects of a method 2300 foroperating a peripheral device (e.g., reference peripheral) in anaugmented workstation environment, according to certain embodiments.Method 2300 can be performed by processing logic that may comprisehardware (circuitry, dedicated logic, etc.), software operating onappropriate hardware (such as a general purpose computing system or adedicated machine), firmware (embedded software), or any combinationthereof. In certain embodiments, method 2300 can be performed by aspectsof computer 1810, reference peripheral 1830(1), display system 1820, ora combination thereof.

At step 2310, method 2300 can include receiving polling data from one ormore sensors (e.g., any of sensors from sensor system 1840), accordingto certain embodiments. The polling data can correspond to physicalcharacteristics of a physical environment around the referenceperipheral. For example, the polling data may indicate where walls,obstructions, other peripheral devices, other user and/or objects arelocated relative to the reference peripheral.

At step 2320, method 2300 can include determining an area to orient avirtual display relative to the reference peripheral within the physicalenvironment based on the physical characteristics. For example, in areaswith close walls or limited space, the virtual display may be displayedcloser to the user and over a relatively small area, as compared to anexpansive area with a large amount of space. In some embodiments, thevirtual display can be configured within zone 3.

At step 2330, method 2300 can include determining a spatial relationshipbetween the reference peripheral and the projected virtual display,according to certain embodiments. For example, the spatial relationshipmay correspond to a fix positional relationship between the referenceperipheral and the virtual display.

At step 2340, method 2300 can include generating control data configuredto cause an AR/VR-based head-mounted display (HMD) to project thevirtual display in the determined area at a maintained spatialrelationship between the peripheral device and the projected virtualdisplay as the peripheral device is moved within the physicalenvironment, as further described above with respect to FIGS. 18-22.

In some embodiments, method 2300 can include detecting that theperipheral device is placed on a surface or interfaced by a user, wherereceiving the polling data from the one or more sensors may occur inresponse to detecting that the peripheral device is placed on thesurface or interfaced by the user. In some cases, method 2300 mayinclude determining that the peripheral device is lifted off of thesurface, and generating second control data to cause the HMD to changethe spatial relationship between the peripheral device and the projectedvirtual display such that a volumetric area occupied by the peripheraldevice and the projected virtual display is reduced. For instance, thismay correspond to zone 1 and/or zone 2, as further described above withrespect to FIGS. 19-21. Method 2300 can further include determining asecond area to orient a virtual interactive display relative to thereference peripheral, where the interactive display is configured tofacilitate an augmentation of functional capabilities of the referenceperipheral. In some cases, method 2300 further includes determining aspatial relationship between the peripheral device and the projectedinteractive display, where the control data is further configured tocause the HMD to project the interactive display in the determinedsecond area and at a maintained spatial relationship between theperipheral device and the projected interactive display as theperipheral device is moved in the physical environment.

In some embodiments, the control data may cause the spatial relationshipbetween the peripheral device and the virtual display to be maintainedsuch that a movement of the peripheral device that is within a thresholddistance from an initial location of the peripheral device does notcause the virtual display to move. The control data may further causethe spatial relationship between the peripheral device and the virtualdisplay to be maintained such that a movement of the peripheral devicethat is greater than the threshold distance from the initial location ofthe peripheral device causes the spatial relationship between theperipheral device and the projected interactive display to be fixed,where the spatial relationship between the peripheral device and theprojected interactive display is fixed as the peripheral device is movedin the physical environment.

In further embodiments, method 2300 can include determining a third areaon the reference peripheral to orient a virtual overlay (e.g., on zone1), where the virtual overlay may be configured to further facilitatethe augmentation of the functional capabilities of the peripheraldevice. Method 2300 can further include determining a spatialrelationship between the peripheral device and the projected virtualoverlay, where the control data is further configured to cause the HMDto project the virtual overlay in the determined third area and at amaintained spatial relationship between the peripheral device and theprojected interactive display as the peripheral device is moved in thephysical environment.

It should be appreciated that the specific steps illustrated in FIG. 23provide a particular method 2300 for operating an augmented workstationenvironment, according to certain embodiments. Other sequences of stepsmay also be performed according to alternative embodiments. Furthermore,additional steps may be added or removed depending on the particularapplications. Any combination of changes can be used and one of ordinaryskill in the art with the benefit of this disclosure would understandthe many variations, modifications, and alternative embodiments thereof.

Peripheral-Centric Augmented Workstation Environment Using a ReferenceHub

In some embodiments, a reference hub (“hub”) 2450 may be configuredwithin an AWE 2400 to interact with peripheral input device(s),2430(1-n), display system 2420, computer 2410, and sensor system 2440,as shown in system 2400 of FIG. 24. Each of computer 2410, displaysystem 2420, sensor system 2440 and peripherals 2430(1-n) may operatesimilarly to their matching counterparts as shown and described withrespect to FIG. 18. As indicated above, sensor system 2440 may be acombination of sensor systems that utilize sensing capabilities ofdisplay system 2420, peripherals 2430(1-n), and hub 2450. Hub 2450 mayperform some or all of the following functions including tracking alocation of some or all of peripherals 2430(1-n) in a physicalenvironment using an independent tracking system, provide enhancedsensing capabilities for high fidelity sensing/tracking of interactionsusing peripherals 2430(1-n), operating as a single communication conduitfor receiving data communications from some or all of peripherals2430(1-n) and relaying said data to computer 1420, display system 2420,or both. Conversely, hub 2450 may operate as a single data conduit torelay communications/control signals from computer 2410 and/or displaysystem 2420 to the one or more peripherals 2430(1-n). Further, hub 2450may be configured to detect an area around one or more peripherals usingan array of on-board sensors to establish zones for content placement inan augmented workstation environment, similar to the referenceperipheral as described above with respect to FIGS. 18-13. Each of thesefunctions are further described below. It should be understood that butfor the inclusion of hub 2450 and its corresponding functionalitydescribed herein, computer 2410, display system 2420, sensor system2440, and peripherals 2430(1-n) may operate similarly as described abovewith respect to FIG. 18 and alternatively, one of ordinary skill in theart with the benefit of this disclosure would understand the manyvariations, modifications, and alternative embodiments thereof.

In some embodiments, display system 2420 (e.g., an HMD, holographicimaging system, augmented reality display, etc.) may track its locationusing a first tracking system, as would be appreciated by one ofordinary skill in the art. Hub 2450 may incorporate a second independenttracking system (e.g., visual tracking, IR, ultrasonic, EM, etc.) totrack a location of some or all of peripherals 2430(1-n) in a physicalenvironment relative to hub 2450. In some cases, hub 2450 may operate asa reference point of origin (e.g., a 000 location in aCartesian-coordinate based system) for tracking peripherals 2430 usingthe second tracking system. The tracked locations of peripherals2430(1-n) using the second tracking system of hub 2450 can be combinedwith (e.g., fused with) the first tracking system of display system2420/computer 2410. Alternatively or additionally, hub 2450 may trackits own location using on-board sensing resources, which can be relayedand combined with the first tracking system. In some cases, hub 2450 mayinclude features (e.g., sensors, LEDs, fiducial mark(s)) for detectingthe location of the hub by the first tracking system. There are certainadvantages to using two tracking systems. For example, the firsttracking system may be limited in terms of accuracy or processingbandwidth such that tracking multiple peripherals may produce poortracking results and/or may slow down system operations, and the firsttracking system may not be accessible or manipulable (e.g., a thirdparty HMD unit) to track peripheral devices in the manner proscribedherein. Other advantages of using a second tracking system are furtheraddressed in the implementations described below.

In some embodiments, hub 2450 may provide enhanced sensing capabilitiesfor high-fidelity sensing and/or tracking of interactions usingperipherals 2430(1-n). Some examples of high-fidelity sensing mayinclude fine precision movements of a user's hand or finger along atouch pad or other sensed region, a movement of a stylus on or near aperipheral, or the like. For example, a small volumetric region adjacentto a peripheral device (e.g., see “zone 2” description above) mayinclude a sensed region that can detect a movement and articulation of ahand or stylus in high fidelity. This capability may not be present instandard display system sensing suites (first tracking system), andtypically not with high fidelity particular to a designated region in aphysical environment. Thus, high precision tracking/sensing can beperformed by one or more peripherals 2430 and/or hub 2450 using a secondtracking/sensor system (e.g., with hub 2450 as the center reference)that may be relayed to and integrated with the first tracking system ofdisplay system 2420.

In certain embodiments, hub 2450 can operate as a single communicationconduit for receiving data communications from some or all ofperipherals 2430(1-n) and relaying said data to computer 1420, displaysystem 2420, or both, and vice versa as described above. Hub 2450 caninteract (e.g., communicate with, transfer data to/from, sharesensor/tracking data, etc.) with each of peripheral devices(1-n) usingseparate communication links (e.g., Bluetooth®, Wi-Fi, IR, ZigBee,Z-Wave, RF, etc.) and coalesce information from peripheral devices 2430to display system 2420 using a single communication link. Thus,significant processing bandwidth may be saved as display system 2420 cancommunicate with one entity (hub 2450) instead of independentlycommunicating with multiple peripheral devices 2430. In someimplementations, hub 2450 can provide capabilities of each peripheraldevice to display system 2420 including the type of peripheral (e.g., akeyboard, mouse, stylus), visual representation information (e.g., howthe peripheral or augmented capabilities of the peripheral may berendered), and the like. That is, hub 2450 may provide renderinginformation for each peripheral device with pre-render datacorresponding to an image of the peripheral, which can then be providedto and imported by display system 2420 without further processing.

In some embodiments, hub 2450 may be a standalone unit, or a part ofanother device including a peripheral (e.g., keyboard, mouse, stylus), asmart device (e.g., phone, watch), a dedicated mini “tower” or the like.Hub 2450 may work across different platforms (e.g., HMD's, Phones,PC's), being agnostic, adapting to various platforms, or mediating andexchanging data between platforms. Hub 2450 may be configured tocommunicate with a variety of operating systems including, but notlimited to, Windows™, Android; IOS; MS-DOS, MacOS, Unix, Linux, etc. Insome cases, multiple hubs may be used together to improve trackingprecision.

In certain implementation, hub 2450 can operate as a point of referenceto detect physical characteristics of a physical environment anddetermine locations to define operating “zones,” similar to a referenceperipheral as further discussed above with respect to FIGS. 19-24. Insuch cases, zones may be highlighted using lighting on hub 2450, orother peripherals, to indicated when a peripheral is placed in atracking zone or not (e.g., for high precision sensing regions, asdescribed above). In some cases, hub 2450 can output indicia of whethera peripheral is in a designated zone or outline zones for the user(e.g., project a box around zone 1, etc.). Further, Hub 2450 may beconfigured to detect and locate peripheral input device(s) and, based onthe detected peripheral device(s), relay corresponding data between thedetected peripheral device(s) 2430(1-n) and display system 2420 and/orcomputer 2420. Hub 2450 may be operable to determine its location(detected from display system 2420 or on-board sensing resources) andother peripherals or interactions with said peripherals with greateraccuracy than provided by display system 2420. In some cases, hub 2450can include a transceiver configured to communicate with a displaysystem for presenting a virtual reality or augmented reality virtualreality environment to a user, a tracking subsystem configured to sensea location of a physical peripheral input device within a physicalenvironment, and one or more processors coupled to the transceiver andthe tracking subsystem, the one or more processors configured todetermine, via the tracking subsystem, the location of the physicalperipheral input device within the physical environment, and transmit,via the transceiver, to the display system, the location of the physicalperipheral input device within the physical environment. In some cases,the tracking subsystem can sense the location of the physical peripheralinput device using a technique different from a technique used by thedisplay system to track a location of the user or a head mounted display(HMD) worn by the user. The technique used by the tracking subsystem totrack the physical peripheral input device and the technique used by thedisplay system to track the location of the user or the HMD may be eachselected from a list comprising: an ultrasonic emitter; an ultrasonicreceiver; a visible light optical sensor; a visible light opticalemitter; a non-visible light optical sensor; a non-visible light opticalemitter; a magnetic field generator; a magnetic field sensor; a radiowave emitter; and a radio wave receiver. In some cases, the hub devicecan further comprising features for determining the location of thedevice within the physical environment by a system used to track alocation of the user or a head mounted display (MHD) worn by the user.The features may be selected from a list including: a sensor; anemitter; and a marking, for example. The sensor can be configured todetect or the emitter is configured to emit: visible light, infraredlight, ultrasound, magnetic fields, or radio waves.

FIG. 25 is a simplified flow chart showing aspects of a method 2500 foroperating a hub to interact with one or more peripheral devices withinan AR/VR workstation environment, according to certain embodiments.Method 2500 can be performed by processing logic that may comprisehardware (circuitry, dedicated logic, etc.), software operating onappropriate hardware (such as a general purpose computing system or adedicated machine), firmware (embedded software), or any combinationthereof. In certain embodiments, method 2500 can be performed by aspectsof hub 2450, computer 2410, display system 1820, or a combinationthereof.

At step 2510, method 2500 can include detecting a physical peripheraldevice within a physical environment. For example, hub 2450 may beconfigured to detect peripheral devices 2430(1-n). The detecting mayinclude determining a presence of a peripheral device via sensor (e.g,vision-based detection), communication (e.g., Bluetooth®, IR), or othersuitable method of determining a presence of a peripheral device, aswould be appreciated by one of ordinary skill in the art with thebenefit of this disclosure.

At step 2520, method 2500 can include determining a location of thephysical peripheral device within the physical environment. This can beperformed, e.g., by hub 2450 using the sensor capabilities describedabove with respect to FIG. 24 and identified as an independent secondtracking system to track peripherals and provide location data,peripheral function data, peripheral visual representation data, and thelike.

At step 2530, method 2500 can include establishing a communicativecoupling with, and receiving data from, the physical peripheral deviceand an AR/VR-based display. That is, hub 2450 can interact (e.g.,communicate with, transfer data to/from, share sensor/tracking data,etc.) with each of peripheral devices(1-n) using separate communicationlinks (e.g., Bluetooth®, Wi-Fi, IR, ZigBee, Z-Wave, RF, etc.) andcoalesce information from peripheral devices 2430 to display system 2420(e.g., HMD, holographic imaging system, AR/VR display system, etc.)using a single communication pathway via hub 2450 to facilitate atransfer of received data between the physical peripheral device and theAR/VR-based display (step 2540).

At step 2550, method 2500 can include determining an area to orient avirtual display relative to the physical peripheral device within thephysical environment based on the determined location of the physicalperipheral device. In some cases, the determine areas may correspond tothe “zones” described above with respect to FIGS. 18-23.

At step 2560, method 2500 can include hub 2450 generating control dataconfigured to cause the AR/VR-based display to project the virtualdisplay in the determined area (e.g., zone 1, 2, 3, 4, etc.). In somecases, the control data may be further configured to cause theAR/VR-based display to maintain a spatial relationship between thephysical peripheral device and the projected virtual display as thephysical peripheral device is moved within the physical environment.

Method 2500 can further include receiving polling data from one or moresensors, where the polling data corresponds to physical characteristicsof a physical environment around the physical peripheral device, andwhere determining the area to orient the virtual display can be furtherbased on the physical characteristics of the physical environment. Insome cases, a movement and location of a head-mounted display (HMD) ofthe AR/VR-based display and the hub can be tracked using a firsttracking system, and the determining a location of the physicalperipheral device within the physical environment can be tracked via thehub using a second tracking system, such that method 2500 may furtherinclude providing data corresponding to the determined location of thephysical peripheral device within the physical environment to the HMD,where the provided data may be configured to cause the integration oftracking via the second tracking system with tracking via the firsttracking system.

In some embodiments, method 2500 may further include detecting a secondphysical peripheral device within the physical environment, determininga location of the second physical peripheral device within the physicalenvironment, establishing a communicative coupling with, and receivingdata from the second physical peripheral device, and coalescing thereceived data from the first and second physical peripheral devices intoaggregate peripheral data,

where the aggregate peripheral data may be transferred to theAR/VR-based display via the hub instead of the individual received datafrom each of the first and second physical peripheral devices. In someembodiments, the hub may be a standalone unit or can be integrated withone of a keyboard, smart device, wearable device, or mini-tower device,or the like.

It should be appreciated that the specific steps illustrated in FIG. 25provide a particular method 2500 for operating a hub to interact withone or more peripheral devices within an AR/VR workstation environment,according to certain embodiments. Other sequences of steps may also beperformed according to alternative embodiments. Furthermore, additionalsteps may be added or removed depending on the particular applications.Any combination of changes can be used and one of ordinary skill in theart with the benefit of this disclosure would understand the manyvariations, modifications, and alternative embodiments thereof.

Modifying an Operation of an Input Device Based on a Tracked Location

In some embodiments, an operation and/or tracking parameter of an inputdevice can be modified based on a tracked location of the input devicein a physical environment. The input device (e.g., a stylus) can betracked to operate within an AR/VR environment (e.g., an augmentedreality work station) and/or a physical environment (e.g., trackingmovement of a stylus along a physical surface, a physical display, orthe like). The input device can be tracked using any suitable 3Dtracking system, similar to those described above with respect to AWE1800 and AWE 2300 above, including but not limited to, 3D locationtracking systems that utilize computer-vision, ultrasound, infra-red,radio-frequency, LIDAR, or other location tracking system, or anycombination thereof. Multiple sensor-based systems that are housed inand/or operated by a display system (e.g., display system 1820/2420(e.g., HMD)), sensor system (e.g., sensor system 1840/2440), a computingdevice (e.g., computer 1810/2410, hub 2450), and/or peripheral device(s)(e.g., peripherals 1830/2430, hub 2450) can be used, as furtherdescribed above with respect to FIGS. 18-25. The input device isdescribed as a stylus in the following embodiments, although it shouldbe understood that any suitable input device can be used in lieu of astylus including a mobile device (e.g., smart phone), computer mouse,presenter device, wearable device (e.g., smart watch), or the like. Insome embodiments, the tracking parameter can include a confidence valueor gain of at least one of the plurality of tracking technologies (e.g.,optical, touch sensing, ultrasonic, etc.) and modifying the parametermay modify an amount that results from one of the tracking technologiesused for tracking the location of the input device by fusing theplurality of tracking technologies. For example, some implementationsmay equally utilize IMU and vision when tracking the input device in oneparticular tracked location, but may switch to 80% IMU vs. 20% opticalwhen tracking the input device in other locations. One of ordinary skillin the art with the benefit of this disclosure would understand the manyvariations, modifications, and alternative embodiments thereof.

The following embodiments (see, e.g., FIGS. 26A-29) describe just someof the myriad ways that the operation of the input device can bemodified based on a tracked location of the input device in a physicalenvironment. For instance, in some implementations, location tracking ofan input device may be configured to operate with a relatively highdegree of accuracy and/or precision when operating on a physical surface(where a user typically has fine hand motor control when their armholding the stylus is supported by a surface), and a comparatively lowerdegree of accuracy and/or precision when operating in free space(“in-air” operation—where a user with no surface support may have areduced capability for fine motor control). In some cases, in-airoperation may be accompanied with higher accuracy measurements when itis detected that the user is supporting their arm holding the inputdevice on a surface, which may be determined via a bio-mechanical modelof a user, vision-based detection, micro-tremor detection, or the like,as further described below at least with respect to FIGS. 26A-26C.

When operating within an AR/VR system, location tracking and/or commandsfrom the input device can be modified depending on the location of theinput device in physical 3D space, as well as the location relative tovirtual objects that populate the virtual/augmented reality environment.For example, greater tracking accuracy may be employed when a user isinterfacing with certain virtual objects in free space (e.g., CADobjects), as further described below at least with respect to FIGS.26A-26C.

In some embodiments, the input device may operate with an increasedlocation tracking sensitivity (e.g., resolution of movement detection)when operating on a 2D surface and/or on a free-floating object in 3Dspace, as opposed to a location tracking sensitivity when operating“in-air” and not directly interfacing physical or virtual objects. Insome cases, tracking on a 2D surface can be supplemented by additionaltracking schemes, such as touch-sensitive sensor arrays (e.g.,capacitive and/or resistive sensors), sensors from other peripheraldevices (e.g., peripherals 2430, hub 2450), display systems (e.g., HMD),or the like, or additional tracking devices on the input device itself(e.g., a tip-mounted sensor on a stylus—such as optical tracking along asurface). In some implementations, certain location tracking systems maybe modified to suspend tracking along certain axes when the input deviceis moving along a 2D surface. For instance, location measurements alongan axis that does not substantially define the contour of the 2D surfacemay be suspended, ignored, or discarded, while the input device isdetermined to be operating along the 2D surface. In further embodiments,tracking precision may be modified based on a proximity of the inputdevice to a display system (e.g., HMD), one or more peripherals (e.g.,keyboard 2430), or the like.

In certain embodiments, other modifications to the input device can beused based on the location of the input device. For example, a powerconsumption of the input device can be modified based on a sensitivitysetting, as described above. Haptic feedback may be employed by theinput device, or by other peripherals or vision systems (e.g., HMD), inresponse to the input device switching interactive modes, such asswitching from “in-air” to 2D surface tracking, switching from “in-air”tracking to interfacing a virtual 3D object, etc.

In further embodiments, certain operations of the input device can bemodified based on its location. For example, some embodiments may usehand writing recognition when the input device is interfacing text on adisplay, virtual display, virtual object, or the like. In some cases,nodes and/or lines/elements on a virtual object may be recognized (e.g.,via AWE 1800) and, in response to certain commands via the input device,the various nodes/lines/elements of the virtual object may “jump” to thelocation of the stylus. Tools, menus, buttons, etc., of the input devicemay be modified based on its location. For instance, CAD tools (e.g.,grab, paint, add a node, delete a node, or other function) may beassigned to certain controls (e.g., buttons) of the input device whenthe input device is interfacing with a virtual object, and otherfunctions may be associated with the input device when operating on a 2Dsurface, or operating “in-air.” In certain embodiments, elementsassociated with an input device may be augmented in an augmented realitysystem in response to tracking the input device at certain locations ina physical environments. For example, an input device feature (e.g.,stylus tip) may be augmented with a virtual addition (e.g., stylus tip)with visible cues (e.g., nib size, width, color, etc.) indicating acurrent function of the input device, which can be modified based on apresent use (e.g., interaction with a surface, virtual object, etc.).Aspects of the invention can include changing a function of one or morebuttons on the input device based on a contextual usage of the inputdevice, changing an operation of the input device in response to theinput device being moved to a same location as a virtual object orphysical object; changing a visual presentation of a virtual feature ofthe input device in response to the input device being moved to a samelocation as the virtual object the physical object, initiating a hapticstimulus by a haptic device coupled to the input device in response tothe input device being moved to a same location as the virtual thephysical object, or the like. These and other features are furtherdescribed below with respect to FIGS. 26A-29. It should be understoodthat said figures are not all encompassing and one of ordinary skill inthe art with the benefit of this disclosure would understand the manyvariations, modifications, and alternative embodiments thereof, andwould understand that any and all of said variations, modifications, andalternative embodiments can be combined with any of the inventiveconcepts presented throughout this disclosure.

As indicated above, an operation of an input device (e.g., stylus) maybe modified based on a location of the input device and/or how the useris interacting with the input device. For example, a tracking accuracyand/or a tracking sensitivity may be modified based on whether the useris operating the input device on a surface (see, e.g., FIG. 26A), “inair,” (see, e.g., FIG. 26B) or “in-air” where the user's arm issupported on a surface (see, e.g., FIG. 26C).

FIG. 26A is a simplified illustration of a user 2610 manipulating avirtual object 2650 displayed “in air” within a physical environment2600, according to certain embodiments. User 2610 is shown manipulatingvirtual object 2650 via an input device 2630 (e.g., stylus). Virtualobject 2650 can be displayed by an HMD (e.g., via system 1800, 2400,etc.), a holographic system, or the like. As indicated above, a usertypically has a relatively low degree of motor control (e.g., accuracyand/or precision) when holding an input device in midair (“in-air”) withno surface support for bracing their arm. As such, 3D tracking accuracyand/or precision can be reduced when the user operates under this typeof “in-air” condition, as a user would typically not be able toarticulate input device 2630 to a level of control provided by highaccuracy/precision location tracking. Increasing/reducing locationtracking accuracy and/or precision can present several advantages. Forexample, a lower accuracy and/or precision measurement may correspond toa lower power consumption for the system (e.g., AWE 1800). Furthermore,reduced tracking accuracy and/or precision may provide a user with amore stable interface. For instance, a very high tracking accuracy maycause an “in-air” interaction to look “shaky” or unstable frommicro-tremors in the user's hand, which tend to be relatively high ascompared to user hand micro-tremors when the user is supporting theirhand, elbow, and/or arm on a surface.

The corresponding system (e.g., AWE 1800) can determine that the user isoperating input device 2630 “in-air” and unsupported by a surface,“in-air” with the user's arm supported on a surface 2660, or on surface2660 by a vision and/or sensor tracking system (e.g., as described abovewith respect to FIGS. 18 and 24), by IMU or other suitable sensingsystem to detect micro-tremors of input device 1630 and/or the user'shand, and/or by a biomechanical model of the user's arm 2640. An IMU maybe configured to detect the micro tremors of the arm. Differentsignatures (e.g., in wavelength and/or amplitude) of responses for thevarious contexts of use can be tracked, and knowing the kinematic chain(e.g., series of movements) allows they system to be able to infer thecontext of the arm from the profile of the movement (e.g., largesweeping movements may be interpreted to mean that the elbow is freelymoving in 3D space). Thus, a size of tremor amplitude, a signature ofthe vibration, tremor frequency, or other parameter, can be used todetermine whether the arm is resting on a surface, suspended in mid-air,or suspended in mid-air and braced against a surface. In some cases,behaviors and movements can be tracked and through machine learning,tracking accuracy can be improved over time based on specific users,applications, contexts, and environments.

A biomechanical model can be a model of a skeletal structure of a userso that knowing a user's position (e.g., location and whether they areseated or standing, etc.) a determination can be made as to whichmovements (e.g., arcs) a stylus would be operated through by the user.Thus, if the stylus is held at a certain orientation or swung through acertain arc, one can glean how the user is holding/using it.

FIG. 26B is a simplified illustration of a user 2610 manipulating avirtual object (a virtual display) 2670 displayed “in air” within aphysical environment 2600 with the user's arm 2640 braced on a surface2660, according to certain embodiments. User 2610 is shown writing on avirtual display 2670 via input device 2630 (e.g., stylus). Virtualdisplay 2670 can be displayed by an HMD (e.g., via system 1800, 2400,etc.), a holographic system, or the like. When a user operates inputdevice “in-air” with their arm (e.g., elbow) supported on a surface, theuser can typically articulate input device 2630 with a higher level ofprecision than they could when their arm is unsupported, as describedabove. As such, 3D tracking accuracy and/or precision can be increased(returned to a relatively higher accuracy and/or precision) when theuser operates under this type of surface-supported “in-air” condition,as a user would typically be able to articulate input device 2630 to alevel of control provided by high accuracy/precision location tracking.As described above, the corresponding system (e.g., AWE 1800) candetermine that the user is operating input device 2630 “in-air” andsupported on surface 2660 by a vision and/or sensor tracking system(e.g., as described above with respect to FIGS. 18 and 24), by IMU orother suitable sensing system to detect micro-tremors of input device1630 and/or the user's hand (micro-tremors may have a lower amplitudewhen the user's arm is supported on a surface), by a biomechanical modelof the user's arm, or the like.

FIG. 26C is a simplified illustration of a user 2610 manipulating avirtual paper 2680 displayed on a surface 2660 within a physicalenvironment 2600, according to certain embodiments. User 2610 is shownvirtually writing on virtual paper 2680 via input device 2630 (e.g.,stylus). Virtual paper 2680 can be displayed by an HMD (e.g., via system1800, 2400, etc.), a holographic system, or the like. When a useroperates input device 2630 on a surface with their arm (e.g., elbow andhand) supported on the surface, the user can typically articulate inputdevice 2630 with a comparatively very high level of precision than theycould when their arm is unsupported, as described above. As such, 3Dtracking accuracy and/or precision can be increased (returned to arelatively higher accuracy and/or precision) when the user operatesunder this type of surface-supported condition, as a user wouldtypically be able to articulate input device 2630 to a level of controlprovided by high accuracy/precision location tracking. As describedabove, the corresponding system (e.g., AWE 1800) can determine that theuser is operating input device 2630 on surface 2660 by a vision and/orsensor tracking system (e.g., as described above with respect to FIGS.18 and 24), by IMU or other suitable sensing system to detectmicro-tremors of input device 1630 and/or the user's hand, by abiomechanical model of the user's arm, or the like.

In some implementations, the location tracking system may be modified tosuspend tracking along certain axes in a Cartesian coordinate systemwhen the input device is moving along surface 2660. For instance,location measurements along an axis that do not substantially define thecontour of the 2D surface may be suspended, ignored, or discarded, whilethe input device is determined to be operating along the 2D surface.

In further embodiments, additional sensor systems may be used toincrease location tracking accuracy and sensitivity when operating alonga surface. For example, additional sensors associated with otherperipheral devices and/or hubs, as described above with respect to FIG.24, can be used to enhance location tracking along a surface.Alternatively or additionally, input device 2630 may include a surfacetracking sensor (e.g., optical sensor on a stylus tip) to track alocation and/or movement along a surface. One of ordinary skill in theart with the benefit of this disclosure would understand the manyvariations, modifications, and alternative embodiments thereof.

FIG. 27 is a simplified flow chart showing aspects of a method 2700 formodifying an operation of an input device based on a detected locationof the input device, according to certain embodiments. Method 2700 canbe performed by processing logic that may comprise hardware (circuitry,dedicated logic, etc.), software operating on appropriate hardware (suchas a general purpose computing system or a dedicated machine), firmware(embedded software), or any combination thereof. In certain embodiments,method 2700 (as well as following methods 2800 and 2900) can beperformed by aspects of computer 1810, reference peripheral 1830(1),display system 1820, or a combination thereof, or using any computersystem described in the present disclosure, as would be understood byone of ordinary skill in the art having the benefit of this disclosure.

At step 2710, method 2700 can include tracking a location of inputdevice 2630 within a physical environment 2600 via a three-dimensional(3D) tracking system, according to certain embodiments.

At step 2720, method 2700 can include determining an operation of inputdevice 2630, which can be used to modify a tracking parameter (e.g.,tracking accuracy) of the 3D tracking system while tracking the locationof the input device based on the determined location of the input devicewithin the physical environment. The operation of input device 2630 caninclude tracked 3D “in-air” operation or 2D surface operation.

At step 2730, in response to determining that the user is operating theinput device “in-air” and not bracing their arm on a surface (step2740), method 2700 can include tracking the location of the input deviceaccording to a first tracking accuracy profile (step 2770), as shown anddescribed above with respect to FIG. 26A.

At step 2730, in response to determining that the user is operating theinput device on a 2D surface, method 2700 can include tracking thelocation of the input device according to a second tracking accuracyprofile (step 2750). In some embodiments, the first tracking accuracyprofile (for “in-air” operations) can have a lower tracking accuracythan a tracking accuracy of the second tracking accuracy profile, asdescribed above with respect to FIG. 26C.

In some implementations, method 2700 include tracking the input deviceusing an additional tracking system while the input device is operatingalong the 2D surface. The additional tracking system can include acapacitance-based touch-sensitive 2D tracking system, a resistance-basedtouch-sensitive 2D tracking system, an image-based tracking system, orthe like. In some instances, the additional 2D tracking system may becoupled to the input device.

In further embodiments, the 3D tracking system that tracks the locationof the input device may track in three axes using a Cartesian coordinatesystem, including a first, second, and third axis. In response todetermining that the input device is operating along the 2D surface,method 2700 can further include determining which of the first, second,and third axes substantially define a contour of the 2D surface andsuspending location tracking along any of the first, second, and thirdaxes that do not substantially define the contour of the 2D surfacewhile the input device is determined to be operating along the 2Dsurface. By suspending location tracking in this manner, powerconsumption may be reduced and spurious measurements that incorrectlyindicate that the device is rising above the 2D surface may be ignored,which can appear as a “smoother” tracking experience to the user.

At step 2730, in response to determining that the user is operating theinput device “in-air” and is bracing their arm on a surface (step 2740),method 2700 can include tracking the location of the input deviceaccording to a third tracking accuracy profile (step 2770), as shown anddescribed above with respect to FIG. 26B. In some cases, the firsttracking accuracy profile can have a lower tracking accuracy than atracking accuracy of the second tracking accuracy profile, the thirdtracking accuracy profile can have a higher tracking accuracy than thetracking accuracy of the second tracking accuracy profile, and the thirdtracking accuracy profile can have a lower tracking accuracy than thetracking accuracy of the second tracking accuracy profile.

In some implementations, determining that the user is bracing their armon a surface may be performed using any of a variety of detectionschemes. For instance, some embodiments of method 2700 can includeanalyzing a real-time biomechanical model of the user's arm that issupporting the input device to determine when the user's arm is bracedon the surface, where the biomechanical model generated via avision-based detection system. Some embodiments of method 2700 caninclude detecting micro-tremors of the input device while the inputdevice is operating “in-air” in 3D space, where micro-tremors at orgreater than a threshold amplitude are indicative of the input deviceoperating “in-air” and the user operating the input device withoutbracing their arm on the surface, and where micro-tremors below thethreshold amplitude are indicative of the input device operating“in-air” and the user operating the input device bracing their arm onthe surface.

It should be appreciated that the specific steps illustrated in FIG. 27provide a particular method 2700 for modifying an operation of an inputdevice based on a detected location of the input device, according tocertain embodiments. Other sequences of steps may also be performedaccording to alternative embodiments. Furthermore, additional steps maybe added or removed depending on the particular applications. Anycombination of changes can be used and one of ordinary skill in the artwith the benefit of this disclosure would understand the manyvariations, modifications, and alternative embodiments thereof.

FIG. 28 is a simplified flow chart showing aspects of a method 2800 formodifying tracking precision of an input device based on a detectedlocation of the input device relative to a peripheral device or avirtual object, according to certain embodiments. Method 2800 can beperformed by processing logic that may comprise hardware (circuitry,dedicated logic, etc.), software operating on appropriate hardware (suchas a general purpose computing system or a dedicated machine), firmware(embedded software), or any combination thereof. In certain embodiments,method 2800 can be performed by aspects of computer 1810, referenceperipheral 1830(1), display system 1820, or a combination thereof.

At step A, continued from method 2700 and step 2710, method 2800 caninclude tracking a location of an input device within a physicalenvironment via a three-dimensional (3D) tracking system. At step 2820,method 2800 can include determining a distance from the input device toa peripheral device or a virtual object. The peripheral device can be aphysical keyboard, computer mouse, controller (e.g., iOT device),display, or any other suitable peripheral device, as would beappreciated by one of ordinary skill in the art. The input device andthe peripheral device are typically separate devices, where both aretypically communicatively coupled to the same computer system (e.g.,system 1800). The virtual object can any virtually rendered image thatcan be interactive, manipulable, and/or accessible by the user.

At step 2820, in response to determining that the peripheral device orvirtual object is within a threshold distance to the input device,method 2800 can include modifying the tracking precision of the inputdevice according to a first tracking sensitivity profile (step 2830).The threshold distance may be set to any suitable value (e.g., 5 mm, 8cm, etc.), as would be appreciated by one of ordinary skill in the artwith the benefit of this disclosure. In some cases, distances may beanywhere from 0 mm (e.g., drawing on or grabbing a real/virtual object),3-5 mm (e.g., virtual airbrush spraying), >10 cm (e.g., indirectlymanipulating a distant object), or other suitable distance.

At step 2820, in response to determining that the peripheral device orvirtual object is not within a threshold distance to the input device,method 2800 can include modifying the tracking precision of the inputdevice according to a second tracking sensitivity profile (step 2840).In some cases, the first tracking sensitivity profile can have a higherprecision measurement than a precision measurement of the secondtracking sensitivity profile.

It should be appreciated that the specific steps illustrated in FIG. 28provide a particular method 2800 for modifying tracking precision of aninput device based on a detected location of the input device relativeto a peripheral device or a virtual object, according to certainembodiments. Other sequences of steps may also be performed according toalternative embodiments. Furthermore, additional steps may be added orremoved depending on the particular applications. Any combination ofchanges can be used and one of ordinary skill in the art with thebenefit of this disclosure would understand the many variations,modifications, and alternative embodiments thereof.

FIG. 29 is a simplified flow chart showing aspects of a method 2900 forchanging an operation of an input device based on its tracked locationwithin a physical environment, according to certain embodiments. Method2900 can be performed by processing logic that may comprise hardware(circuitry, dedicated logic, etc.), software operating on appropriatehardware (such as a general purpose computing system or a dedicatedmachine), firmware (embedded software), or any combination thereof. Incertain embodiments, method 2900 can be performed by aspects of computer1810, reference peripheral 1830(1), display system 1820, or acombination thereof.

At step 2910, method 2900 can include tracking a location of an inputdevice within a physical environment via a three-dimensional (3D)tracking system.

At step 2920, method 2900 can include tracking the input device withinan augmented reality or virtual reality (AR/VR) environment based on thetracked location of the input device within the physical environment.

At step 2930, method 2900 can include changing an operation of the inputdevice based on the tracked location of the input device within thephysical environment. For example, some embodiments may initiate handwriting recognition when the input device is interfacing with text on adisplay, a virtual display, a virtual object, or the like. In somecases, nodes and/or lines/elements on a virtual object may be recognized(e.g., via AWE 1800) and, in response to certain commands via the inputdevice, the various nodes/lines/elements of the virtual object may“jump” to the location of the stylus. Tools, menus, buttons, etc., ofthe input device may be modified based on its location. For instance,CAD tools (e.g., grab, paint, add a node, delete a node, or otherfunction) may be assigned to certain controls (e.g., buttons) of theinput device when the input device is interfacing with a virtual object,and other functions may be associated with the input device whenoperating on a 2D surface, or operating “in-air.” In certainembodiments, elements associated with an input device may be augmentedin an augmented reality system in response to tracking the input deviceat certain locations in a physical environments. For example, an inputdevice feature (e.g., stylus tip) may be augmented with a virtualaddition (e.g., stylus tip) with visible cues (e.g., nib size, width,color, etc.) indicating a current function of the input device, whichcan be modified based on a present use (e.g., interaction with asurface, virtual object, etc.).

It should be appreciated that the specific steps illustrated in FIG. 29provide a particular method 2900 for changing an operation of an inputdevice based on its tracked location within a physical environment,according to certain embodiments. Other sequences of steps may also beperformed according to alternative embodiments. Furthermore, additionalsteps may be added or removed depending on the particular applications.Any combination of changes can be used and one of ordinary skill in theart with the benefit of this disclosure would understand the manyvariations, modifications, and alternative embodiments thereof.

It will be appreciated that any of the disclosed methods (orcorresponding apparatuses, programs, data carriers, etc.) may be carriedout by either a host or client, depending on the specific implementation(i.e. the disclosed methods/apparatuses are a form of communication(s),and as such, may be carried out from either ‘point of view’, i.e., incorresponding to each other fashion).

As used in this specification, any formulation used of the style “atleast one of A, B or C”, and the formulation “at least one of A, B andC” use a disjunctive “or” and a disjunctive “and” such that thoseformulations comprise any and all joint and several permutations of A,B, C, that is, A alone, B alone, C alone, A and B in any order, A and Cin any order, B and C in any order and A, B, C in any order. There maybe more or less than three features used in such formulations.

In the claims, any reference signs placed between parentheses shall notbe construed as limiting the claim. The word ‘comprising’ does notexclude the presence of other elements or steps then those listed in aclaim. Furthermore, the terms “a” or “an,” as used herein, are definedas one or more than one. Also, the use of introductory phrases such as“at least one” and “one or more” in the claims should not be construedto imply that the introduction of another claim element by theindefinite articles “a” or “an” limits any particular claim containingsuch introduced claim element to inventions containing only one suchelement, even when the same claim includes the introductory phrases “oneor more” or “at least one” and indefinite articles such as “a” or “an.”The same holds true for the use of definite articles. Unless statedotherwise, terms such as “first” and “second” are used to arbitrarilydistinguish between the elements such terms describe. Thus, these termsare not necessarily intended to indicate temporal or otherprioritization of such elements. The mere fact that certain measures arerecited in mutually different claims does not indicate that acombination of these measures cannot be used to advantage.

Unless otherwise explicitly stated as incompatible, or the physics orotherwise of the embodiments, example or claims prevent such acombination, the features of the foregoing embodiments and examples, andof the following claims may be integrated together in any suitablearrangement, especially ones where there is a beneficial effect in doingso. This is not limited to only any specified benefit, and instead mayarise from an “ex post facto” benefit. This is to say that thecombination of features is not limited by the described forms,particularly the form (e.g. numbering) of the example(s), embodiment(s),or dependency of the claim(s). Moreover, this also applies to the phrase“in one embodiment”, “according to an embodiment” and the like, whichare merely a stylistic form of wording and are not to be construed aslimiting the following features to a separate embodiment to all otherinstances of the same or similar wording. This is to say, a reference to‘an’, ‘one’ or ‘some’ embodiment(s) may be a reference to any one ormore, and/or all embodiments, or combination(s) thereof, disclosed.Also, similarly, the reference to “the” embodiment may not be limited tothe immediately preceding embodiment.

Certain figures in this specification are flow charts illustratingmethods and systems. It will be understood that each block of these flowcharts, and combinations of blocks in these flow charts, may beimplemented by computer program instructions. These computer programinstructions may be loaded onto a computer or other programmableapparatus to produce a machine, such that the instructions which executeon the computer or other programmable apparatus create structures forimplementing the functions specified in the flow chart block or blocks.These computer program instructions may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable apparatus to function in a particular manner, such that theinstructions stored in the computer-readable memory produce an articleof manufacture including instruction structures which implement thefunction specified in the flow chart block or blocks. The computerprogram instructions may also be loaded onto a computer or otherprogrammable apparatus to cause a series of operational steps to beperformed on the computer or other programmable apparatus to produce acomputer implemented process such that the instructions which execute onthe computer or other programmable apparatus provide steps forimplementing the functions specified in the flow chart block or blocks.Accordingly, blocks of the flow charts support combinations ofstructures for performing the specified functions and combinations ofsteps for performing the specified functions. It will also be understoodthat each block of the flow charts, and combinations of blocks in theflow charts, can be implemented by special purpose hardware-basedcomputer systems which perform the specified functions or steps, orcombinations of special purpose hardware and computer instructions.

For example, any number of computer programming languages, such as C,C++, C# (CSharp), Perl, Ada, Python, Pascal, Small Talk, FORTRAN,assembly language, and the like, may be used to implement machineinstructions. Further, various programming approaches such asprocedural, object-oriented or artificial intelligence techniques may beemployed, depending on the requirements of each particularimplementation. Compiler programs and/or virtual machine programsexecuted by computer systems generally translate higher levelprogramming languages to generate sets of machine instructions that maybe executed by one or more processors to perform a programmed functionor set of function

The foregoing description of one or more implementations providesillustration and description, but is not intended to be exhaustive or tolimit the scope of the invention to the precise form disclosed.Modifications and variations are possible in light of the aboveteachings or may be acquired from practice of various implementations ofthe present disclosure.

What is claimed is:
 1. A system comprising one or more processorsconfigured to: track a location of a stylus device within a physicalenvironment via a three-dimensional (3D) tracking system, wherein thestylus device is coupled to a virtual reality display system and whereinthe tracking the location of the location of the stylus device is usedfor interacting with the virtual reality display system; and modify atracking parameter of the 3D tracking system while tracking the locationof the stylus device based on the tracked location of the stylus devicewithin the physical environment, wherein modifying the trackingparameter of the stylus device further includes: tracking the locationof the stylus device according to a first tracking accuracy profile inresponse to determining that the stylus device is at a locationindicative of the stylus device operating “in-air” in 3D space; trackingthe location of the stylus device according to a second trackingaccuracy profile in response to determining that the stylus device isoperating along a two-dimensional (2D) surface; and tracking thelocation of the stylus device according to a third accuracy trackingprofile in response to determining that the stylus device is at alocation indicative of a user operating the stylus device “in-air” in 3Dspace and bracing their arm against a surface.
 2. The system of claim 1wherein the tracking parameter is further modified based on a biometricmodel of a user of the stylus device.
 3. The system of claim 2 whereinthe biometric model is used by the one or more processors to determine alocation of the stylus device corresponding to the user holding thestylus device with an appendage.
 4. The system of claim 1 wherein thetwo-dimensional surface is a physical surface.
 5. The system of claim 1wherein the first tracking accuracy profile has a lower trackingaccuracy than a tracking accuracy of the second tracking accuracyprofile.
 6. The system of claim 1 wherein the first tracking accuracyprofile has a lower tracking accuracy than a tracking accuracy of thesecond tracking accuracy profile, wherein the third tracking accuracyprofile has a higher tracking accuracy than the tracking accuracy of thesecond tracking accuracy profile, and wherein the third trackingaccuracy profile has a lower tracking accuracy than the trackingaccuracy of the second tracking accuracy profile.
 7. The system of claim1 wherein determining that the stylus device is at a location indicativeof a user operating the stylus device “in-air” in 3D space and bracingtheir arm against a surface includes at least one of: using an imagingsensor to detect an orientation of the stylus device or the user; usinga biomechanical model of the user to determine a location indicative ofa portion of user being in a determined position while operating thestylus device; or detecting micro-tremors of the stylus device inducedby the user while the stylus device is operated, wherein micro-tremorsat or greater than a threshold amplitude are indicative of the stylusdevice operating “in-air” and the user operating the stylus devicewithout bracing their arm against the surface, and wherein micro-tremorsbelow the threshold amplitude are indicative of the stylus deviceoperating “in-air” and the user operating the stylus device bracingtheir arm against the surface.
 8. The system of claim 1 wherein thestylus device is of a list including: a stylus; a mobile device; acomputer mouse; a presenter device; and a wearable device.
 9. The systemof claim 1 wherein the 3D tracking system tracks in three axes in aCartesian coordinate system including a first, second, and third axis,and wherein in response to determining that the stylus device isoperating along the 2D surface, the one or more processors are furtherconfigured to: determine which of the first, second, and third axessubstantially define a contour of the 2D surface; and suspend trackingalong any of the first, second, and third axes that does notsubstantially define the contour of the 2D surface while the stylusdevice is determined to be operating along the 2D surface.
 10. Thesystem of claim 1 wherein the one or more processors are configured to:determine a distance from the stylus device to a peripheral device, thestylus device and the peripheral device being separate andcommunicatively coupled to the same computer system; and modify atracking precision of the stylus device while tracking the location ofthe stylus device based on the determined distance from the stylusdevice to the peripheral device.
 11. The system of claim 10 wherein theone or more processors are configured to: modify the tracking precisionof the stylus device according to a first tracking sensitivity profilein response to determining that the stylus device is within a thresholddistance from the peripheral device; and modify the tracking precisionof the stylus device according to a second tracking sensitivity profilein response to determining that the stylus device is not within athreshold distance from the peripheral device.
 12. The system of claim11 wherein the first tracking sensitivity profile has a higher precisionmeasurement than a precision measurement of the second trackingsensitivity profile.
 13. The system of claim 1 wherein the one or moreprocessors are configured to: change an operation of the stylus devicebased on the tracked location of the stylus device within the physicalenvironment.
 14. The system of claim 1 wherein modifying the trackingparameter of the stylus device based on the tracked location of thestylus device within the physical environment causes an operationincluding one of a list of operations including: changing a function ofone or more buttons on the stylus device based on a contextual usage ofthe stylus device; changing an operation of the stylus device inresponse to the stylus device being moved to a same location as avirtual object or physical object; changing a visual presentation of avirtual feature of the stylus device in response to the stylus devicebeing moved to a same location as the virtual object the physicalobject; and initiating a haptic stimulus by a haptic device coupled tothe stylus device in response to the stylus device being moved to a samelocation as the virtual the physical object.
 15. A computer implementedmethod comprising: tracking a location of an stylus device within aphysical environment via a three-dimensional (3D) tracking system,wherein the stylus device is coupled to a virtual reality display systemand wherein the tracking the location of the location of the stylusdevice is used for interacting with the virtual reality display system;and modifying a tracking parameter of the 3D tracking system whiletracking the location of the stylus device based on the tracked locationof the stylus device within the physical environment, wherein modifyingthe tracking parameter of the stylus device further includes: trackingthe location of the stylus device according to a first tracking accuracyprofile in response to determining that the stylus device is at alocation indicative of the stylus device operating “in-air” in 3D space;tracking the location of the stylus device according to a secondtracking accuracy profile in response to determining that the stylusdevice is operating along a two-dimensional (2D) surface; and trackingthe location of the stylus device according to a third accuracy trackingprofile in response to determining that the stylus device is at alocation indicative of a user operating the stylus device “in-air” in 3Dspace and bracing their arm against a surface.
 16. The system of claim15 wherein the tracking parameter is further modified based on abiometric model of a user of the stylus device.
 17. The system of claim16 wherein the biometric model is used to determine a location of thestylus device corresponding to the user holding the stylus device withan appendage.
 18. The system of claim 15 wherein the two-dimensionalsurface is a physical surface.
 19. The system of claim 15 wherein thefirst tracking accuracy profile has a lower tracking accuracy than atracking accuracy of the second tracking accuracy profile.
 20. Thesystem of claim 15 wherein the first tracking accuracy profile has alower tracking accuracy than a tracking accuracy of the second trackingaccuracy profile, wherein the third tracking accuracy profile has ahigher tracking accuracy than the tracking accuracy of the secondtracking accuracy profile, and wherein the third tracking accuracyprofile has a lower tracking accuracy than the tracking accuracy of thesecond tracking accuracy profile.
 21. The system of claim 15 whereindetermining that the stylus device is at a location indicative of a useroperating the stylus device “in-air” in 3D space and bracing their armagainst a surface includes at least one of: using an imaging sensor todetect an orientation of the stylus device or the user; using abiomechanical model of the user to determine a location indicative of aportion of user being in a determined position while operating thestylus device; or detecting micro-tremors of the stylus device inducedby the user while the stylus device is operated, wherein micro-tremorsat or greater than a threshold amplitude are indicative of the stylusdevice operating “in-air” and the user operating the stylus devicewithout bracing their arm against the surface, and wherein micro-tremorsbelow the threshold amplitude are indicative of the stylus deviceoperating “in-air” and the user operating the stylus device bracingtheir arm against the surface.